2024-01-15T11:14:17.424 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-15T11:14:17.425 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} --archive /home/teuthworker/archive/rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517064 --name rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi --verbose --owner scheduled_rishabh@teuthology -- /home/teuthworker/archive/rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517064/orig.config.yaml 2024-01-15T11:14:17.467 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-15T11:14:17.543 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517064 branch: rishabh-clone-stats description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} email: null first_in_suite: false job_id: '7517064' kernel: client: sha1: distro flavor: default kdb: true sha1: c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: rishabh-clone-stats ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: mgr-vol-clone-stats-temp sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 9877 sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b sleep_before_teardown: 0 suite: fs:volumes suite_branch: mgr-vol-clone-stats-temp suite_path: /home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa suite_relpath: qa suite_repo: https://github.com/rishabh-d-dave/ceph suite_sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi113.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLCA/CGEJo18u9e0CDj25sBHw8iC1BuaeVgBQ5ju1ZHJYeitElGPXNnh/2Fwdt41tcUkHPs+17DXH3op1xm/0xM= smithi141.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8dkBsd2zlYWrvj7QYzhElnvOE/iuNKCyODK6mvmlcRLi0HkL8ocHk57GidyxP8BMY8jl31lLozx7XyO3GSprE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestCloneProgress teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-15_11:07:32 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-15T11:14:17.543 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa; will attempt to use it 2024-01-15T11:14:17.544 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks 2024-01-15T11:14:17.544 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-15T11:14:17.546 INFO:teuthology.task.internal:Checking packages... 2024-01-15T11:14:17.565 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '95c90f413c4bb65e3e56909855f2308fd9bda05b' 2024-01-15T11:14:17.566 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-15T11:14:17.566 INFO:teuthology.packaging:ref: None 2024-01-15T11:14:17.566 INFO:teuthology.packaging:tag: None 2024-01-15T11:14:17.566 INFO:teuthology.packaging:branch: rishabh-clone-stats 2024-01-15T11:14:17.566 INFO:teuthology.packaging:sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T11:14:17.566 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=rishabh-clone-stats 2024-01-15T11:14:17.732 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7629.g95c90f41 2024-01-15T11:14:17.733 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-15T11:14:17.739 INFO:teuthology.task.internal:no buildpackages task found 2024-01-15T11:14:17.740 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-15T11:14:17.759 INFO:teuthology.task.internal:Saving configuration 2024-01-15T11:14:17.776 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-15T11:14:17.782 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-15T11:14:17.807 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517064', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 11:08:30.429485', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLCA/CGEJo18u9e0CDj25sBHw8iC1BuaeVgBQ5ju1ZHJYeitElGPXNnh/2Fwdt41tcUkHPs+17DXH3op1xm/0xM='} 2024-01-15T11:14:17.832 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi141.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517064', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 11:08:30.427461', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8dkBsd2zlYWrvj7QYzhElnvOE/iuNKCyODK6mvmlcRLi0HkL8ocHk57GidyxP8BMY8jl31lLozx7XyO3GSprE='} 2024-01-15T11:14:17.832 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-15T11:14:17.838 INFO:teuthology.task.internal:roles: ubuntu@smithi113.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-01-15T11:14:17.838 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-01-15T11:14:17.838 INFO:teuthology.run_tasks:Running task console_log... 2024-01-15T11:14:17.896 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1b515a3160>, signals=[15]) 2024-01-15T11:14:17.896 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-15T11:14:17.902 INFO:teuthology.task.internal:Opening connections... 2024-01-15T11:14:17.902 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi113.front.sepia.ceph.com 2024-01-15T11:14:17.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T11:14:17.976 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi141.front.sepia.ceph.com 2024-01-15T11:14:17.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T11:14:18.046 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-15T11:14:18.053 DEBUG:teuthology.orchestra.run.smithi113:> uname -m 2024-01-15T11:14:18.072 INFO:teuthology.orchestra.run.smithi113.stdout:x86_64 2024-01-15T11:14:18.073 DEBUG:teuthology.orchestra.run.smithi113:> cat /etc/os-release 2024-01-15T11:14:18.131 INFO:teuthology.orchestra.run.smithi113.stdout:NAME="Red Hat Enterprise Linux" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION="8.6 (Ootpa)" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:ID="rhel" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:ID_LIKE="fedora" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION_ID="8.6" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:PLATFORM_ID="platform:el8" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:ANSI_COLOR="0;31" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:HOME_URL="https://www.redhat.com/" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-15T11:14:18.132 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-15T11:14:18.133 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-15T11:14:18.133 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-15T11:14:18.133 INFO:teuthology.lock.ops:Updating smithi113.front.sepia.ceph.com on lock server 2024-01-15T11:14:18.160 DEBUG:teuthology.orchestra.run.smithi141:> uname -m 2024-01-15T11:14:18.182 INFO:teuthology.orchestra.run.smithi141.stdout:x86_64 2024-01-15T11:14:18.182 DEBUG:teuthology.orchestra.run.smithi141:> cat /etc/os-release 2024-01-15T11:14:18.242 INFO:teuthology.orchestra.run.smithi141.stdout:NAME="Red Hat Enterprise Linux" 2024-01-15T11:14:18.242 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION="8.6 (Ootpa)" 2024-01-15T11:14:18.242 INFO:teuthology.orchestra.run.smithi141.stdout:ID="rhel" 2024-01-15T11:14:18.242 INFO:teuthology.orchestra.run.smithi141.stdout:ID_LIKE="fedora" 2024-01-15T11:14:18.242 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_ID="8.6" 2024-01-15T11:14:18.242 INFO:teuthology.orchestra.run.smithi141.stdout:PLATFORM_ID="platform:el8" 2024-01-15T11:14:18.242 INFO:teuthology.orchestra.run.smithi141.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-15T11:14:18.242 INFO:teuthology.orchestra.run.smithi141.stdout:ANSI_COLOR="0;31" 2024-01-15T11:14:18.243 INFO:teuthology.orchestra.run.smithi141.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-15T11:14:18.243 INFO:teuthology.orchestra.run.smithi141.stdout:HOME_URL="https://www.redhat.com/" 2024-01-15T11:14:18.243 INFO:teuthology.orchestra.run.smithi141.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-15T11:14:18.243 INFO:teuthology.orchestra.run.smithi141.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-15T11:14:18.243 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:14:18.243 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-15T11:14:18.243 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-15T11:14:18.243 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-15T11:14:18.243 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-15T11:14:18.243 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server 2024-01-15T11:14:18.260 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-15T11:14:18.269 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-15T11:14:18.275 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-15T11:14:18.275 DEBUG:teuthology.orchestra.run.smithi113:> test '!' -e /home/ubuntu/cephtest 2024-01-15T11:14:18.277 DEBUG:teuthology.orchestra.run.smithi141:> test '!' -e /home/ubuntu/cephtest 2024-01-15T11:14:18.301 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-15T11:14:18.307 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-15T11:14:18.307 DEBUG:teuthology.orchestra.run.smithi113:> test -z $(ls -A /var/lib/ceph) 2024-01-15T11:14:18.336 DEBUG:teuthology.orchestra.run.smithi141:> test -z $(ls -A /var/lib/ceph) 2024-01-15T11:14:18.371 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-15T11:14:18.425 INFO:teuthology.run_tasks:Running task kernel... 2024-01-15T11:14:18.441 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': 'c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b'} 2024-01-15T11:14:18.441 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-01-15T11:14:18.441 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-01-15T11:14:18.441 DEBUG:teuthology.orchestra.run.smithi113:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-15T11:14:18.442 DEBUG:teuthology.orchestra.run.smithi141:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-15T11:14:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T11:14:18.461 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2024-01-15T11:14:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T11:14:18.462 DEBUG:teuthology.orchestra.run.smithi141:> uname -r 2024-01-15T11:14:18.520 INFO:teuthology.orchestra.run.smithi113.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-15T11:14:18.521 INFO:teuthology.task.kernel:Running kernel on smithi113: 4.18.0-513.9.1.el8_9.x86_64 2024-01-15T11:14:18.521 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum install -y kernel 2024-01-15T11:14:18.522 INFO:teuthology.orchestra.run.smithi141.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-15T11:14:18.522 INFO:teuthology.task.kernel:Running kernel on smithi141: 4.18.0-513.9.1.el8_9.x86_64 2024-01-15T11:14:18.523 DEBUG:teuthology.orchestra.run.smithi141:> sudo yum install -y kernel 2024-01-15T11:14:20.013 INFO:teuthology.orchestra.run.smithi113.stdout:Updating Subscription Management repositories. 2024-01-15T11:14:20.013 INFO:teuthology.orchestra.run.smithi113.stdout:Unable to read consumer identity 2024-01-15T11:14:20.065 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2024-01-15T11:14:20.065 INFO:teuthology.orchestra.run.smithi141.stdout:Unable to read consumer identity 2024-01-15T11:14:20.136 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:14:20.137 INFO:teuthology.orchestra.run.smithi113.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T11:14:20.137 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:14:20.188 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:14:20.188 INFO:teuthology.orchestra.run.smithi141.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T11:14:20.189 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:14:21.862 INFO:teuthology.orchestra.run.smithi113.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2024-01-15T11:14:21.947 INFO:teuthology.orchestra.run.smithi141.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-01-15T11:14:23.173 INFO:teuthology.orchestra.run.smithi141.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-01-15T11:14:23.621 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 16 MB 00:01 2024-01-15T11:14:26.764 INFO:teuthology.orchestra.run.smithi141.stdout:lab-extras 343 kB/s | 24 kB 00:00 2024-01-15T11:14:27.171 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 476 kB/s | 24 kB 00:00 2024-01-15T11:14:27.542 INFO:teuthology.orchestra.run.smithi141.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-15T11:14:27.543 INFO:teuthology.orchestra.run.smithi141.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-15T11:14:27.633 INFO:teuthology.orchestra.run.smithi141.stdout:Dependencies resolved. 2024-01-15T11:14:27.635 INFO:teuthology.orchestra.run.smithi141.stdout:Nothing to do. 2024-01-15T11:14:27.635 INFO:teuthology.orchestra.run.smithi141.stdout:Complete! 2024-01-15T11:14:27.784 DEBUG:teuthology.orchestra.run.smithi141:> echo no | sudo yum reinstall kernel || true 2024-01-15T11:14:27.948 INFO:teuthology.orchestra.run.smithi113.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-15T11:14:27.949 INFO:teuthology.orchestra.run.smithi113.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-15T11:14:28.027 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-01-15T11:14:28.028 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-01-15T11:14:28.029 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-01-15T11:14:28.121 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2024-01-15T11:14:28.121 INFO:teuthology.orchestra.run.smithi141.stdout:Unable to read consumer identity 2024-01-15T11:14:28.155 DEBUG:teuthology.orchestra.run.smithi113:> echo no | sudo yum reinstall kernel || true 2024-01-15T11:14:28.181 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:14:28.181 INFO:teuthology.orchestra.run.smithi141.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T11:14:28.182 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:14:28.492 INFO:teuthology.orchestra.run.smithi113.stdout:Updating Subscription Management repositories. 2024-01-15T11:14:28.492 INFO:teuthology.orchestra.run.smithi113.stdout:Unable to read consumer identity 2024-01-15T11:14:28.496 INFO:teuthology.orchestra.run.smithi141.stdout:Last metadata expiration check: 0:00:02 ago on Mon 15 Jan 2024 11:14:26 AM UTC. 2024-01-15T11:14:28.548 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:14:28.548 INFO:teuthology.orchestra.run.smithi113.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T11:14:28.548 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:14:28.564 INFO:teuthology.orchestra.run.smithi141.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-15T11:14:28.579 INFO:teuthology.orchestra.run.smithi141.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-15T11:14:28.619 INFO:teuthology.orchestra.run.smithi141.stderr:Error: No packages marked for reinstall. 2024-01-15T11:14:28.667 DEBUG:teuthology.orchestra.run.smithi141:> sudo yum reinstall -y kernel || true 2024-01-15T11:14:28.856 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:00:01 ago on Mon 15 Jan 2024 11:14:27 AM UTC. 2024-01-15T11:14:28.935 INFO:teuthology.orchestra.run.smithi113.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-15T11:14:28.959 INFO:teuthology.orchestra.run.smithi113.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-15T11:14:29.001 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2024-01-15T11:14:29.001 INFO:teuthology.orchestra.run.smithi141.stdout:Unable to read consumer identity 2024-01-15T11:14:29.005 INFO:teuthology.orchestra.run.smithi113.stderr:Error: No packages marked for reinstall. 2024-01-15T11:14:29.052 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum reinstall -y kernel || true 2024-01-15T11:14:29.056 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:14:29.056 INFO:teuthology.orchestra.run.smithi141.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T11:14:29.056 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:14:29.363 INFO:teuthology.orchestra.run.smithi141.stdout:Last metadata expiration check: 0:00:03 ago on Mon 15 Jan 2024 11:14:26 AM UTC. 2024-01-15T11:14:29.385 INFO:teuthology.orchestra.run.smithi113.stdout:Updating Subscription Management repositories. 2024-01-15T11:14:29.386 INFO:teuthology.orchestra.run.smithi113.stdout:Unable to read consumer identity 2024-01-15T11:14:29.424 INFO:teuthology.orchestra.run.smithi141.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-15T11:14:29.424 INFO:teuthology.orchestra.run.smithi141.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-15T11:14:29.440 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:14:29.440 INFO:teuthology.orchestra.run.smithi113.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T11:14:29.440 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:14:29.458 INFO:teuthology.orchestra.run.smithi141.stderr:Error: No packages marked for reinstall. 2024-01-15T11:14:29.505 DEBUG:teuthology.orchestra.run.smithi141:> rpm -q kernel | sort -rV | head -n 1 2024-01-15T11:14:29.661 INFO:teuthology.orchestra.run.smithi141.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-15T11:14:29.661 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-15T11:14:29.661 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-15T11:14:29.662 DEBUG:teuthology.parallel:result is None 2024-01-15T11:14:29.745 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:00:02 ago on Mon 15 Jan 2024 11:14:27 AM UTC. 2024-01-15T11:14:29.812 INFO:teuthology.orchestra.run.smithi113.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-15T11:14:29.812 INFO:teuthology.orchestra.run.smithi113.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-15T11:14:29.868 INFO:teuthology.orchestra.run.smithi113.stderr:Error: No packages marked for reinstall. 2024-01-15T11:14:29.915 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q kernel | sort -rV | head -n 1 2024-01-15T11:14:30.079 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-15T11:14:30.080 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-15T11:14:30.080 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-15T11:14:30.080 DEBUG:teuthology.parallel:result is None 2024-01-15T11:14:30.080 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-15T11:14:30.089 INFO:teuthology.task.internal:Creating test directory... 2024-01-15T11:14:30.089 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-15T11:14:30.092 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-15T11:14:30.111 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-15T11:14:30.118 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-15T11:14:30.125 INFO:teuthology.task.internal:Creating archive directory... 2024-01-15T11:14:30.125 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-15T11:14:30.149 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-15T11:14:30.187 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-15T11:14:30.194 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-15T11:14:30.194 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-15T11:14:30.215 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-15T11:14:30.264 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T11:14:30.269 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T11:14:30.279 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T11:14:30.290 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T11:14:30.292 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-15T11:14:30.299 INFO:teuthology.task.internal:Configuring sudo... 2024-01-15T11:14:30.299 DEBUG:teuthology.orchestra.run.smithi113:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-15T11:14:30.302 DEBUG:teuthology.orchestra.run.smithi141:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-15T11:14:30.331 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-15T11:14:30.368 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-15T11:14:30.369 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-15T11:14:30.371 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-15T11:14:30.389 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T11:14:30.473 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T11:14:30.542 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T11:14:30.586 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T11:14:30.628 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:14:30.628 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-15T11:14:30.695 DEBUG:teuthology.orchestra.run.smithi141:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T11:14:30.741 DEBUG:teuthology.orchestra.run.smithi141:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T11:14:30.798 DEBUG:teuthology.orchestra.run.smithi141:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T11:14:30.883 DEBUG:teuthology.orchestra.run.smithi141:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T11:14:30.927 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:14:30.927 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-15T11:14:30.995 DEBUG:teuthology.orchestra.run.smithi113:> sudo service rsyslog restart 2024-01-15T11:14:30.997 DEBUG:teuthology.orchestra.run.smithi141:> sudo service rsyslog restart 2024-01-15T11:14:31.043 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-15T11:14:31.080 INFO:teuthology.orchestra.run.smithi141.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-15T11:14:31.557 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-15T11:14:31.566 INFO:teuthology.task.internal:Starting timer... 2024-01-15T11:14:31.566 INFO:teuthology.run_tasks:Running task pcp... 2024-01-15T11:14:31.576 INFO:teuthology.run_tasks:Running task selinux... 2024-01-15T11:14:31.585 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-15T11:14:31.585 DEBUG:teuthology.orchestra.run.smithi113:> sudo service auditd rotate 2024-01-15T11:14:31.633 INFO:teuthology.orchestra.run.smithi113.stdout:Rotating logs: 2024-01-15T11:14:31.635 DEBUG:teuthology.orchestra.run.smithi141:> sudo service auditd rotate 2024-01-15T11:14:31.694 INFO:teuthology.orchestra.run.smithi141.stdout:Rotating logs: 2024-01-15T11:14:31.696 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-15T11:14:31.696 DEBUG:teuthology.orchestra.run.smithi113:> /usr/sbin/getenforce 2024-01-15T11:14:31.721 INFO:teuthology.orchestra.run.smithi113.stdout:Permissive 2024-01-15T11:14:31.721 DEBUG:teuthology.orchestra.run.smithi141:> /usr/sbin/getenforce 2024-01-15T11:14:31.745 INFO:teuthology.orchestra.run.smithi141.stdout:Permissive 2024-01-15T11:14:31.745 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi113.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi141.front.sepia.ceph.com': 'permissive'} 2024-01-15T11:14:31.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-15T11:14:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T11:14:31.789 DEBUG:teuthology.orchestra.run.smithi141:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-15T11:14:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T11:14:31.818 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-15T11:14:31.818 DEBUG:teuthology.orchestra.run.smithi113:> sudo /usr/sbin/setenforce permissive 2024-01-15T11:14:31.857 DEBUG:teuthology.orchestra.run.smithi141:> sudo /usr/sbin/setenforce permissive 2024-01-15T11:14:31.890 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-15T11:14:31.900 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-15T11:14:31.907 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-15T11:14:32.071 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-15T11:14:32.087 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-15T11:14:32.089 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi113.front.sepia.ceph.com,smithi141.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-15T11:21:31.791 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi113.front.sepia.ceph.com'), Remote(name='ubuntu@smithi141.front.sepia.ceph.com')] 2024-01-15T11:21:31.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T11:21:31.794 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T11:21:31.866 DEBUG:teuthology.orchestra.run.smithi113:> true 2024-01-15T11:21:31.932 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T11:21:31.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-15T11:21:31.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T11:21:32.001 DEBUG:teuthology.orchestra.run.smithi141:> true 2024-01-15T11:21:32.026 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-15T11:21:32.026 INFO:teuthology.run_tasks:Running task clock... 2024-01-15T11:21:32.037 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-15T11:21:32.037 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-15T11:21:32.037 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T11:21:32.039 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-15T11:21:32.039 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T11:21:32.068 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-15T11:21:32.083 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-15T11:21:32.097 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-15T11:21:32.111 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-15T11:21:32.153 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: ntpd: command not found 2024-01-15T11:21:32.164 INFO:teuthology.orchestra.run.smithi113.stdout:506 Cannot talk to daemon 2024-01-15T11:21:32.176 INFO:teuthology.orchestra.run.smithi141.stderr:sudo: ntpd: command not found 2024-01-15T11:21:32.179 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-15T11:21:32.188 INFO:teuthology.orchestra.run.smithi141.stdout:506 Cannot talk to daemon 2024-01-15T11:21:32.194 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-15T11:21:32.203 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-15T11:21:32.217 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-15T11:21:32.261 INFO:teuthology.orchestra.run.smithi113.stderr:bash: ntpq: command not found 2024-01-15T11:21:32.266 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-15T11:21:32.266 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-01-15T11:21:32.266 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T11:21:32.266 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T11:21:32.266 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T11:21:32.266 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T11:21:32.300 INFO:teuthology.orchestra.run.smithi141.stderr:bash: ntpq: command not found 2024-01-15T11:21:32.304 INFO:teuthology.orchestra.run.smithi141.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-15T11:21:32.304 INFO:teuthology.orchestra.run.smithi141.stdout:=============================================================================== 2024-01-15T11:21:32.304 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T11:21:32.304 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T11:21:32.304 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T11:21:32.304 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T11:21:32.304 INFO:teuthology.run_tasks:Running task install... 2024-01-15T11:21:32.312 DEBUG:teuthology.task.install:project ceph 2024-01-15T11:21:32.313 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}} 2024-01-15T11:21:32.313 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'} 2024-01-15T11:21:32.313 INFO:teuthology.task.install:Using flavor: default 2024-01-15T11:21:32.318 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-15T11:21:32.318 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-15T11:21:32.318 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'tag': None, 'wait_for_package': False} 2024-01-15T11:21:32.319 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T11:21:32.321 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'tag': None, 'wait_for_package': False} 2024-01-15T11:21:32.321 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T11:21:32.462 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/ 2024-01-15T11:21:32.462 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g95c90f41 2024-01-15T11:21:32.529 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/ 2024-01-15T11:21:32.529 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g95c90f41 2024-01-15T11:21:32.662 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-15T11:21:32.662 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:21:32.662 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-15T11:21:32.691 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-15T11:21:32.692 DEBUG:teuthology.orchestra.run.smithi113:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/95c90f413c4bb65e3e56909855f2308fd9bda05b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-15T11:21:32.696 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-15T11:21:32.696 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:21:32.696 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-15T11:21:32.725 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-15T11:21:32.725 DEBUG:teuthology.orchestra.run.smithi141:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/95c90f413c4bb65e3e56909855f2308fd9bda05b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-15T11:21:32.761 DEBUG:teuthology.orchestra.run.smithi113:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-15T11:21:32.794 DEBUG:teuthology.orchestra.run.smithi141:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-15T11:21:32.838 DEBUG:teuthology.orchestra.run.smithi113:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-15T11:21:32.873 DEBUG:teuthology.orchestra.run.smithi141:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-15T11:21:32.905 INFO:teuthology.orchestra.run.smithi113.stdout:check_obsoletes = 1 2024-01-15T11:21:32.906 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum clean all 2024-01-15T11:21:32.940 INFO:teuthology.orchestra.run.smithi141.stdout:check_obsoletes = 1 2024-01-15T11:21:32.941 DEBUG:teuthology.orchestra.run.smithi141:> sudo yum clean all 2024-01-15T11:21:33.281 INFO:teuthology.orchestra.run.smithi113.stdout:Updating Subscription Management repositories. 2024-01-15T11:21:33.318 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2024-01-15T11:21:33.622 INFO:teuthology.orchestra.run.smithi113.stdout:57 files removed 2024-01-15T11:21:33.639 INFO:teuthology.orchestra.run.smithi141.stdout:57 files removed 2024-01-15T11:21:33.669 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-15T11:21:33.685 DEBUG:teuthology.orchestra.run.smithi141:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-15T11:21:34.003 INFO:teuthology.orchestra.run.smithi113.stdout:Updating Subscription Management repositories. 2024-01-15T11:21:34.020 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2024-01-15T11:21:34.903 INFO:teuthology.orchestra.run.smithi113.stdout:ceph packages for x86_64 174 kB/s | 81 kB 00:00 2024-01-15T11:21:35.024 INFO:teuthology.orchestra.run.smithi141.stdout:ceph packages for x86_64 143 kB/s | 81 kB 00:00 2024-01-15T11:21:35.424 INFO:teuthology.orchestra.run.smithi141.stdout:ceph noarch packages 26 kB/s | 9.5 kB 00:00 2024-01-15T11:21:35.446 INFO:teuthology.orchestra.run.smithi113.stdout:ceph noarch packages 19 kB/s | 9.5 kB 00:00 2024-01-15T11:21:35.855 INFO:teuthology.orchestra.run.smithi113.stdout:ceph source packages 3.9 kB/s | 1.5 kB 00:00 2024-01-15T11:21:35.867 INFO:teuthology.orchestra.run.smithi141.stdout:ceph source packages 3.6 kB/s | 1.5 kB 00:00 2024-01-15T11:21:36.147 INFO:teuthology.orchestra.run.smithi113.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2024-01-15T11:21:36.184 INFO:teuthology.orchestra.run.smithi141.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-01-15T11:21:38.156 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 16 MB 00:01 2024-01-15T11:21:39.868 INFO:teuthology.orchestra.run.smithi141.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 16 MB 00:03 2024-01-15T11:21:41.781 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 401 kB/s | 24 kB 00:00 2024-01-15T11:21:42.174 INFO:teuthology.orchestra.run.smithi113.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 28 MB/s | 8.8 MB 00:00 2024-01-15T11:21:43.485 INFO:teuthology.orchestra.run.smithi141.stdout:lab-extras 402 kB/s | 24 kB 00:00 2024-01-15T11:21:44.620 INFO:teuthology.orchestra.run.smithi141.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 MB/s | 65 MB 00:01 2024-01-15T11:21:44.784 INFO:teuthology.orchestra.run.smithi113.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 MB/s | 65 MB 00:01 2024-01-15T11:21:54.395 INFO:teuthology.orchestra.run.smithi141.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 26 MB/s | 8.8 MB 00:00 2024-01-15T11:21:55.267 INFO:teuthology.orchestra.run.smithi113.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 62 MB/s | 58 MB 00:00 2024-01-15T11:21:57.146 INFO:teuthology.orchestra.run.smithi141.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 57 MB/s | 58 MB 00:01 2024-01-15T11:22:10.712 INFO:teuthology.orchestra.run.smithi113.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-15T11:22:10.713 INFO:teuthology.orchestra.run.smithi113.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-15T11:22:10.731 INFO:teuthology.orchestra.run.smithi113.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-15T11:22:10.732 INFO:teuthology.orchestra.run.smithi113.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-15T11:22:10.734 INFO:teuthology.orchestra.run.smithi113.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-15T11:22:10.735 INFO:teuthology.orchestra.run.smithi113.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-15T11:22:10.738 INFO:teuthology.orchestra.run.smithi113.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-15T11:22:10.742 INFO:teuthology.orchestra.run.smithi113.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-15T11:22:10.743 INFO:teuthology.orchestra.run.smithi113.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-15T11:22:10.949 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-01-15T11:22:10.963 INFO:teuthology.orchestra.run.smithi113.stdout:=================================================================================================================================== 2024-01-15T11:22:10.963 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-01-15T11:22:10.963 INFO:teuthology.orchestra.run.smithi113.stdout:=================================================================================================================================== 2024-01-15T11:22:10.964 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-01-15T11:22:10.964 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-15T11:22:10.964 INFO:teuthology.orchestra.run.smithi113.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-15T11:22:10.964 INFO:teuthology.orchestra.run.smithi113.stdout: ceph x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 6.4 k 2024-01-15T11:22:10.964 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 5.4 M 2024-01-15T11:22:10.964 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 920 k 2024-01-15T11:22:10.964 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 171 k 2024-01-15T11:22:10.964 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 1.6 M 2024-01-15T11:22:10.964 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 133 k 2024-01-15T11:22:10.964 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 1.8 M 2024-01-15T11:22:10.964 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 7.4 M 2024-01-15T11:22:10.965 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 50 k 2024-01-15T11:22:10.965 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 14 M 2024-01-15T11:22:10.965 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 49 M 2024-01-15T11:22:10.965 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 262 k 2024-01-15T11:22:10.965 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 412 k 2024-01-15T11:22:10.965 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 263 k 2024-01-15T11:22:10.965 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-top noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 18 k 2024-01-15T11:22:10.965 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-15T11:22:10.965 INFO:teuthology.orchestra.run.smithi113.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-15T11:22:10.965 INFO:teuthology.orchestra.run.smithi113.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-15T11:22:10.965 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 31 k 2024-01-15T11:22:10.965 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 839 k 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 139 k 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 197 k 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 370 k 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 364 k 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 107 k 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 94 k 2024-01-15T11:22:10.966 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 3.6 M 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 181 k 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrading: 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: librados2 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 3.8 M 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 3.7 M 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-15T11:22:10.967 INFO:teuthology.orchestra.run.smithi113.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 23 M 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 28 k 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 2.3 M 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 246 k 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 4.6 M 2024-01-15T11:22:10.968 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 18 M 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 15 k 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 24 k 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 183 k 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-01-15T11:22:10.969 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-15T11:22:10.970 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-15T11:22:10.970 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 517 k 2024-01-15T11:22:10.970 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-15T11:22:10.970 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 6.9 M 2024-01-15T11:22:10.970 INFO:teuthology.orchestra.run.smithi113.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-15T11:22:10.970 INFO:teuthology.orchestra.run.smithi113.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-15T11:22:10.970 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-15T11:22:10.970 INFO:teuthology.orchestra.run.smithi113.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-15T11:22:10.970 INFO:teuthology.orchestra.run.smithi113.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-15T11:22:10.970 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-15T11:22:10.970 INFO:teuthology.orchestra.run.smithi113.stdout: lua-devel x86_64 5.3.4-12.el8 codeready-builder-for-rhel-8-x86_64-rpms 30 k 2024-01-15T11:22:10.970 INFO:teuthology.orchestra.run.smithi113.stdout: luarocks noarch 3.5.0-1.el8 epel 167 k 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 45 k 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 128 k 2024-01-15T11:22:10.971 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-15T11:22:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-15T11:22:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-15T11:22:10.974 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout:Enabling module streams: 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql 10 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout:=================================================================================================================================== 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout:Install 131 Packages 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrade 16 Packages 2024-01-15T11:22:10.975 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:22:10.976 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 218 M 2024-01-15T11:22:10.976 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-01-15T11:22:11.166 INFO:teuthology.orchestra.run.smithi113.stdout:(1/147): ceph-18.0.0-7629.g95c90f41.el8.x86_64. 42 kB/s | 6.4 kB 00:00 2024-01-15T11:22:11.467 INFO:teuthology.orchestra.run.smithi113.stdout:(2/147): ceph-fuse-18.0.0-7629.g95c90f41.el8.x8 3.0 MB/s | 920 kB 00:00 2024-01-15T11:22:11.533 INFO:teuthology.orchestra.run.smithi113.stdout:(3/147): ceph-immutable-object-cache-18.0.0-762 2.5 MB/s | 171 kB 00:00 2024-01-15T11:22:11.609 INFO:teuthology.orchestra.run.smithi113.stdout:(4/147): ceph-base-18.0.0-7629.g95c90f41.el8.x8 9.0 MB/s | 5.4 MB 00:00 2024-01-15T11:22:11.843 INFO:teuthology.orchestra.run.smithi113.stdout:(5/147): ceph-mds-18.0.0-7629.g95c90f41.el8.x86 7.5 MB/s | 2.3 MB 00:00 2024-01-15T11:22:11.876 INFO:teuthology.orchestra.run.smithi113.stdout:(6/147): ceph-mgr-18.0.0-7629.g95c90f41.el8.x86 6.1 MB/s | 1.6 MB 00:00 2024-01-15T11:22:12.481 INFO:teuthology.orchestra.run.smithi141.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-15T11:22:12.482 INFO:teuthology.orchestra.run.smithi141.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-15T11:22:12.500 INFO:teuthology.orchestra.run.smithi141.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-15T11:22:12.501 INFO:teuthology.orchestra.run.smithi141.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-15T11:22:12.503 INFO:teuthology.orchestra.run.smithi141.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-15T11:22:12.504 INFO:teuthology.orchestra.run.smithi141.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-15T11:22:12.507 INFO:teuthology.orchestra.run.smithi141.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-15T11:22:12.511 INFO:teuthology.orchestra.run.smithi141.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-15T11:22:12.512 INFO:teuthology.orchestra.run.smithi141.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-15T11:22:12.534 INFO:teuthology.orchestra.run.smithi113.stdout:(7/147): ceph-mon-18.0.0-7629.g95c90f41.el8.x86 6.7 MB/s | 4.6 MB 00:00 2024-01-15T11:22:12.720 INFO:teuthology.orchestra.run.smithi141.stdout:Dependencies resolved. 2024-01-15T11:22:12.734 INFO:teuthology.orchestra.run.smithi141.stdout:=================================================================================================================================== 2024-01-15T11:22:12.734 INFO:teuthology.orchestra.run.smithi141.stdout: Package Arch Version Repository Size 2024-01-15T11:22:12.734 INFO:teuthology.orchestra.run.smithi141.stdout:=================================================================================================================================== 2024-01-15T11:22:12.734 INFO:teuthology.orchestra.run.smithi141.stdout:Installing: 2024-01-15T11:22:12.734 INFO:teuthology.orchestra.run.smithi141.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-15T11:22:12.734 INFO:teuthology.orchestra.run.smithi141.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: ceph x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 6.4 k 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-base x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 5.4 M 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-fuse x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 920 k 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 171 k 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 1.6 M 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 133 k 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 1.8 M 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 7.4 M 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 50 k 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 14 M 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-test x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 49 M 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-volume noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 262 k 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: cephadm noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 412 k 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 263 k 2024-01-15T11:22:12.735 INFO:teuthology.orchestra.run.smithi141.stdout: cephfs-top noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 18 k 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 31 k 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: libcephfs2 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 839 k 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: librados-devel x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 139 k 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-01-15T11:22:12.736 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cephfs x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 197 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rados x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 370 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rbd x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 364 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rgw x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 107 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-fuse x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 94 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-mirror x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 3.6 M 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-nbd x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 181 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout:Upgrading: 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: librados2 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 3.8 M 2024-01-15T11:22:12.737 INFO:teuthology.orchestra.run.smithi141.stdout: librbd1 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 3.7 M 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout:Installing dependencies: 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-common x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 23 M 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 28 k 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mds x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 2.3 M 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 246 k 2024-01-15T11:22:12.738 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mon x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 4.6 M 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-osd x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 18 M 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 15 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-selinux x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 24 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: libcephsqlite x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 183 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-15T11:22:12.739 INFO:teuthology.orchestra.run.smithi141.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 517 k 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: librgw2 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 6.9 M 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: lua-devel x86_64 5.3.4-12.el8 codeready-builder-for-rhel-8-x86_64-rpms 30 k 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: luarocks noarch 3.5.0-1.el8 epel 167 k 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-15T11:22:12.740 INFO:teuthology.orchestra.run.smithi141.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 45 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 128 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-15T11:22:12.741 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-15T11:22:12.742 INFO:teuthology.orchestra.run.smithi141.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-15T11:22:12.743 INFO:teuthology.orchestra.run.smithi141.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-15T11:22:12.744 INFO:teuthology.orchestra.run.smithi141.stdout:Enabling module streams: 2024-01-15T11:22:12.745 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql 10 2024-01-15T11:22:12.745 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:22:12.745 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction Summary 2024-01-15T11:22:12.745 INFO:teuthology.orchestra.run.smithi141.stdout:=================================================================================================================================== 2024-01-15T11:22:12.745 INFO:teuthology.orchestra.run.smithi141.stdout:Install 131 Packages 2024-01-15T11:22:12.745 INFO:teuthology.orchestra.run.smithi141.stdout:Upgrade 16 Packages 2024-01-15T11:22:12.745 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:22:12.745 INFO:teuthology.orchestra.run.smithi141.stdout:Total download size: 218 M 2024-01-15T11:22:12.745 INFO:teuthology.orchestra.run.smithi141.stdout:Downloading Packages: 2024-01-15T11:22:13.113 INFO:teuthology.orchestra.run.smithi141.stdout:(1/147): ceph-18.0.0-7629.g95c90f41.el8.x86_64. 20 kB/s | 6.4 kB 00:00 2024-01-15T11:22:13.351 INFO:teuthology.orchestra.run.smithi113.stdout:(8/147): ceph-osd-18.0.0-7629.g95c90f41.el8.x86 12 MB/s | 18 MB 00:01 2024-01-15T11:22:13.468 INFO:teuthology.orchestra.run.smithi113.stdout:(9/147): ceph-selinux-18.0.0-7629.g95c90f41.el8 208 kB/s | 24 kB 00:00 2024-01-15T11:22:13.547 INFO:teuthology.orchestra.run.smithi141.stdout:(2/147): ceph-fuse-18.0.0-7629.g95c90f41.el8.x8 2.1 MB/s | 920 kB 00:00 2024-01-15T11:22:13.928 INFO:teuthology.orchestra.run.smithi141.stdout:(3/147): ceph-base-18.0.0-7629.g95c90f41.el8.x8 4.7 MB/s | 5.4 MB 00:01 2024-01-15T11:22:14.072 INFO:teuthology.orchestra.run.smithi141.stdout:(4/147): ceph-immutable-object-cache-18.0.0-762 325 kB/s | 171 kB 00:00 2024-01-15T11:22:14.160 INFO:teuthology.orchestra.run.smithi113.stdout:(10/147): ceph-common-18.0.0-7629.g95c90f41.el8 7.2 MB/s | 23 MB 00:03 2024-01-15T11:22:14.297 INFO:teuthology.orchestra.run.smithi141.stdout:(5/147): ceph-mds-18.0.0-7629.g95c90f41.el8.x86 6.3 MB/s | 2.3 MB 00:00 2024-01-15T11:22:14.319 INFO:teuthology.orchestra.run.smithi113.stdout:(11/147): cephfs-mirror-18.0.0-7629.g95c90f41.e 1.6 MB/s | 263 kB 00:00 2024-01-15T11:22:14.444 INFO:teuthology.orchestra.run.smithi113.stdout:(12/147): libcephfs-devel-18.0.0-7629.g95c90f41 248 kB/s | 31 kB 00:00 2024-01-15T11:22:14.786 INFO:teuthology.orchestra.run.smithi113.stdout:(13/147): libcephfs2-18.0.0-7629.g95c90f41.el8. 2.4 MB/s | 839 kB 00:00 2024-01-15T11:22:14.928 INFO:teuthology.orchestra.run.smithi113.stdout:(14/147): ceph-radosgw-18.0.0-7629.g95c90f41.el 5.8 MB/s | 14 MB 00:02 2024-01-15T11:22:14.929 INFO:teuthology.orchestra.run.smithi141.stdout:(6/147): ceph-mgr-18.0.0-7629.g95c90f41.el8.x86 1.9 MB/s | 1.6 MB 00:00 2024-01-15T11:22:14.953 INFO:teuthology.orchestra.run.smithi113.stdout:(15/147): libcephsqlite-18.0.0-7629.g95c90f41.e 1.1 MB/s | 183 kB 00:00 2024-01-15T11:22:15.106 INFO:teuthology.orchestra.run.smithi141.stdout:(7/147): ceph-mon-18.0.0-7629.g95c90f41.el8.x86 5.7 MB/s | 4.6 MB 00:00 2024-01-15T11:22:15.470 INFO:teuthology.orchestra.run.smithi113.stdout:(16/147): librados-devel-18.0.0-7629.g95c90f41. 256 kB/s | 139 kB 00:00 2024-01-15T11:22:15.629 INFO:teuthology.orchestra.run.smithi113.stdout:(17/147): libradosstriper1-18.0.0-7629.g95c90f4 766 kB/s | 517 kB 00:00 2024-01-15T11:22:15.746 INFO:teuthology.orchestra.run.smithi113.stdout:(18/147): python3-ceph-argparse-18.0.0-7629.g95 384 kB/s | 45 kB 00:00 2024-01-15T11:22:15.863 INFO:teuthology.orchestra.run.smithi113.stdout:(19/147): python3-ceph-common-18.0.0-7629.g95c9 1.1 MB/s | 128 kB 00:00 2024-01-15T11:22:15.988 INFO:teuthology.orchestra.run.smithi113.stdout:(20/147): python3-cephfs-18.0.0-7629.g95c90f41. 1.5 MB/s | 197 kB 00:00 2024-01-15T11:22:16.206 INFO:teuthology.orchestra.run.smithi113.stdout:(21/147): python3-rados-18.0.0-7629.g95c90f41.e 1.7 MB/s | 370 kB 00:00 2024-01-15T11:22:16.397 INFO:teuthology.orchestra.run.smithi113.stdout:(22/147): python3-rbd-18.0.0-7629.g95c90f41.el8 1.9 MB/s | 364 kB 00:00 2024-01-15T11:22:16.506 INFO:teuthology.orchestra.run.smithi113.stdout:(23/147): python3-rgw-18.0.0-7629.g95c90f41.el8 983 kB/s | 107 kB 00:00 2024-01-15T11:22:16.614 INFO:teuthology.orchestra.run.smithi113.stdout:(24/147): rbd-fuse-18.0.0-7629.g95c90f41.el8.x8 866 kB/s | 94 kB 00:00 2024-01-15T11:22:17.469 INFO:teuthology.orchestra.run.smithi141.stdout:(8/147): ceph-radosgw-18.0.0-7629.g95c90f41.el8 5.9 MB/s | 14 MB 00:02 2024-01-15T11:22:17.544 INFO:teuthology.orchestra.run.smithi141.stdout:(9/147): ceph-selinux-18.0.0-7629.g95c90f41.el8 325 kB/s | 24 kB 00:00 2024-01-15T11:22:17.565 INFO:teuthology.orchestra.run.smithi113.stdout:(25/147): librgw2-18.0.0-7629.g95c90f41.el8.x86 3.3 MB/s | 6.9 MB 00:02 2024-01-15T11:22:17.773 INFO:teuthology.orchestra.run.smithi113.stdout:(26/147): rbd-nbd-18.0.0-7629.g95c90f41.el8.x86 872 kB/s | 181 kB 00:00 2024-01-15T11:22:17.907 INFO:teuthology.orchestra.run.smithi113.stdout:(27/147): ceph-grafana-dashboards-18.0.0-7629.g 209 kB/s | 28 kB 00:00 2024-01-15T11:22:18.032 INFO:teuthology.orchestra.run.smithi113.stdout:(28/147): ceph-mgr-cephadm-18.0.0-7629.g95c90f4 1.0 MB/s | 133 kB 00:00 2024-01-15T11:22:18.557 INFO:teuthology.orchestra.run.smithi113.stdout:(29/147): ceph-test-18.0.0-7629.g95c90f41.el8.x 9.7 MB/s | 49 MB 00:05 2024-01-15T11:22:18.641 INFO:teuthology.orchestra.run.smithi113.stdout:(30/147): rbd-mirror-18.0.0-7629.g95c90f41.el8. 1.8 MB/s | 3.6 MB 00:02 2024-01-15T11:22:18.699 INFO:teuthology.orchestra.run.smithi113.stdout:(31/147): ceph-mgr-dashboard-18.0.0-7629.g95c90 2.6 MB/s | 1.8 MB 00:00 2024-01-15T11:22:18.841 INFO:teuthology.orchestra.run.smithi113.stdout:(32/147): ceph-mgr-rook-18.0.0-7629.g95c90f41.e 354 kB/s | 50 kB 00:00 2024-01-15T11:22:18.866 INFO:teuthology.orchestra.run.smithi113.stdout:(33/147): ceph-mgr-modules-core-18.0.0-7629.g95 1.1 MB/s | 246 kB 00:00 2024-01-15T11:22:18.917 INFO:teuthology.orchestra.run.smithi113.stdout:(34/147): ceph-prometheus-alerts-18.0.0-7629.g9 199 kB/s | 15 kB 00:00 2024-01-15T11:22:18.967 INFO:teuthology.orchestra.run.smithi113.stdout:(35/147): ceph-volume-18.0.0-7629.g95c90f41.el8 2.5 MB/s | 262 kB 00:00 2024-01-15T11:22:19.059 INFO:teuthology.orchestra.run.smithi113.stdout:(36/147): cephfs-top-18.0.0-7629.g95c90f41.el8. 198 kB/s | 18 kB 00:00 2024-01-15T11:22:19.176 INFO:teuthology.orchestra.run.smithi113.stdout:(37/147): ceph-mgr-diskprediction-local-18.0.0- 12 MB/s | 7.4 MB 00:00 2024-01-15T11:22:19.201 INFO:teuthology.orchestra.run.smithi113.stdout:(38/147): cephadm-18.0.0-7629.g95c90f41.el8.noa 1.4 MB/s | 412 kB 00:00 2024-01-15T11:22:19.290 INFO:teuthology.orchestra.run.smithi141.stdout:(10/147): ceph-osd-18.0.0-7629.g95c90f41.el8.x8 4.1 MB/s | 18 MB 00:04 2024-01-15T11:22:19.357 INFO:teuthology.orchestra.run.smithi141.stdout:(11/147): cephfs-mirror-18.0.0-7629.g95c90f41.e 3.9 MB/s | 263 kB 00:00 2024-01-15T11:22:19.391 INFO:teuthology.orchestra.run.smithi141.stdout:(12/147): libcephfs-devel-18.0.0-7629.g95c90f41 931 kB/s | 31 kB 00:00 2024-01-15T11:22:19.499 INFO:teuthology.orchestra.run.smithi141.stdout:(13/147): libcephfs2-18.0.0-7629.g95c90f41.el8. 7.6 MB/s | 839 kB 00:00 2024-01-15T11:22:19.518 INFO:teuthology.orchestra.run.smithi113.stdout:(39/147): liboath-2.6.2-3.el8.x86_64.rpm 172 kB/s | 59 kB 00:00 2024-01-15T11:22:19.549 INFO:teuthology.orchestra.run.smithi141.stdout:(14/147): libcephsqlite-18.0.0-7629.g95c90f41.e 3.6 MB/s | 183 kB 00:00 2024-01-15T11:22:19.602 INFO:teuthology.orchestra.run.smithi113.stdout:(40/147): libunwind-1.3.1-3.el8.x86_64.rpm 187 kB/s | 75 kB 00:00 2024-01-15T11:22:19.625 INFO:teuthology.orchestra.run.smithi141.stdout:(15/147): librados-devel-18.0.0-7629.g95c90f41. 1.8 MB/s | 139 kB 00:00 2024-01-15T11:22:19.717 INFO:teuthology.orchestra.run.smithi141.stdout:(16/147): libradosstriper1-18.0.0-7629.g95c90f4 5.5 MB/s | 517 kB 00:00 2024-01-15T11:22:19.719 INFO:teuthology.orchestra.run.smithi113.stdout:(41/147): gperftools-libs-2.7-9.el8.x86_64.rpm 464 kB/s | 306 kB 00:00 2024-01-15T11:22:19.761 INFO:teuthology.orchestra.run.smithi113.stdout:(42/147): luarocks-3.5.0-1.el8.noarch.rpm 689 kB/s | 167 kB 00:00 2024-01-15T11:22:19.828 INFO:teuthology.orchestra.run.smithi113.stdout:(43/147): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 401 kB/s | 44 kB 00:00 2024-01-15T11:22:19.928 INFO:teuthology.orchestra.run.smithi113.stdout:(44/147): python3-beautifulsoup4-4.6.3-2.el8.1. 1.1 MB/s | 185 kB 00:00 2024-01-15T11:22:19.962 INFO:teuthology.orchestra.run.smithi113.stdout:(45/147): python3-cachetools-3.1.1-4.el8.noarch 246 kB/s | 33 kB 00:00 2024-01-15T11:22:19.995 INFO:teuthology.orchestra.run.smithi113.stdout:(46/147): python3-asyncssh-2.7.0-2.el8.noarch.r 1.2 MB/s | 476 kB 00:00 2024-01-15T11:22:20.021 INFO:teuthology.orchestra.run.smithi113.stdout:(47/147): python3-certifi-2018.10.15-7.el8.noar 176 kB/s | 16 kB 00:00 2024-01-15T11:22:20.096 INFO:teuthology.orchestra.run.smithi113.stdout:(48/147): python3-cheroot-8.5.2-1.el8.noarch.rp 1.3 MB/s | 173 kB 00:00 2024-01-15T11:22:20.138 INFO:teuthology.orchestra.run.smithi113.stdout:(49/147): python3-cherrypy-18.4.0-1.el8.noarch. 2.6 MB/s | 384 kB 00:00 2024-01-15T11:22:20.155 INFO:teuthology.orchestra.run.smithi113.stdout:(50/147): python3-google-auth-1.1.1-10.el8.noar 607 kB/s | 81 kB 00:00 2024-01-15T11:22:20.188 INFO:teuthology.orchestra.run.smithi113.stdout:(51/147): python3-jaraco-6.2-6.el8.noarch.rpm 120 kB/s | 11 kB 00:00 2024-01-15T11:22:20.230 INFO:teuthology.orchestra.run.smithi113.stdout:(52/147): python3-jaraco-functools-2.0-4.el8.no 191 kB/s | 18 kB 00:00 2024-01-15T11:22:20.297 INFO:teuthology.orchestra.run.smithi113.stdout:(53/147): python3-logutils-0.3.5-11.el8.noarch. 448 kB/s | 49 kB 00:00 2024-01-15T11:22:20.340 INFO:teuthology.orchestra.run.smithi113.stdout:(54/147): python3-more-itertools-7.2.0-3.el8.no 540 kB/s | 59 kB 00:00 2024-01-15T11:22:20.390 INFO:teuthology.orchestra.run.smithi113.stdout:(55/147): python3-kubernetes-11.0.0-6.el8.noarc 5.0 MB/s | 1.2 MB 00:00 2024-01-15T11:22:20.415 INFO:teuthology.orchestra.run.smithi113.stdout:(56/147): python3-natsort-7.1.1-2.el8.noarch.rp 513 kB/s | 60 kB 00:00 2024-01-15T11:22:20.450 INFO:teuthology.orchestra.run.smithi141.stdout:(17/147): librgw2-18.0.0-7629.g95c90f41.el8.x86 9.4 MB/s | 6.9 MB 00:00 2024-01-15T11:22:20.490 INFO:teuthology.orchestra.run.smithi113.stdout:(57/147): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2024-01-15T11:22:20.501 INFO:teuthology.orchestra.run.smithi141.stdout:(18/147): python3-ceph-argparse-18.0.0-7629.g95 899 kB/s | 45 kB 00:00 2024-01-15T11:22:20.541 INFO:teuthology.orchestra.run.smithi113.stdout:(58/147): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-01-15T11:22:20.551 INFO:teuthology.orchestra.run.smithi141.stdout:(19/147): python3-ceph-common-18.0.0-7629.g95c9 2.5 MB/s | 128 kB 00:00 2024-01-15T11:22:20.566 INFO:teuthology.orchestra.run.smithi113.stdout:(59/147): python3-repoze-lru-0.7-6.el8.noarch.r 223 kB/s | 34 kB 00:00 2024-01-15T11:22:20.599 INFO:teuthology.orchestra.run.smithi113.stdout:(60/147): python3-routes-2.4.1-12.el8.noarch.rp 1.8 MB/s | 196 kB 00:00 2024-01-15T11:22:20.609 INFO:teuthology.orchestra.run.smithi141.stdout:(20/147): python3-cephfs-18.0.0-7629.g95c90f41. 3.3 MB/s | 197 kB 00:00 2024-01-15T11:22:20.641 INFO:teuthology.orchestra.run.smithi113.stdout:(61/147): python3-rsa-4.9-2.el8.noarch.rpm 641 kB/s | 64 kB 00:00 2024-01-15T11:22:20.667 INFO:teuthology.orchestra.run.smithi113.stdout:(62/147): python3-simplegeneric-0.8.1-17.el8.no 186 kB/s | 19 kB 00:00 2024-01-15T11:22:20.676 INFO:teuthology.orchestra.run.smithi141.stdout:(21/147): python3-rados-18.0.0-7629.g95c90f41.e 5.4 MB/s | 370 kB 00:00 2024-01-15T11:22:20.700 INFO:teuthology.orchestra.run.smithi113.stdout:(63/147): python3-singledispatch-3.4.0.3-18.el8 238 kB/s | 24 kB 00:00 2024-01-15T11:22:20.742 INFO:teuthology.orchestra.run.smithi113.stdout:(64/147): python3-tempora-1.14.1-5.el8.noarch.r 291 kB/s | 29 kB 00:00 2024-01-15T11:22:20.752 INFO:teuthology.orchestra.run.smithi141.stdout:(22/147): python3-rbd-18.0.0-7629.g95c90f41.el8 4.8 MB/s | 364 kB 00:00 2024-01-15T11:22:20.767 INFO:teuthology.orchestra.run.smithi113.stdout:(65/147): python3-trustme-0.6.0-4.el8.noarch.rp 274 kB/s | 27 kB 00:00 2024-01-15T11:22:20.804 INFO:teuthology.orchestra.run.smithi141.stdout:(23/147): python3-rgw-18.0.0-7629.g95c90f41.el8 2.0 MB/s | 107 kB 00:00 2024-01-15T11:22:20.818 INFO:teuthology.orchestra.run.smithi113.stdout:(66/147): python3-waitress-1.4.3-1.el8.noarch.r 2.1 MB/s | 255 kB 00:00 2024-01-15T11:22:20.854 INFO:teuthology.orchestra.run.smithi141.stdout:(24/147): rbd-fuse-18.0.0-7629.g95c90f41.el8.x8 1.8 MB/s | 94 kB 00:00 2024-01-15T11:22:20.876 INFO:teuthology.orchestra.run.smithi113.stdout:(67/147): python3-websocket-client-0.56.0-5.el8 564 kB/s | 61 kB 00:00 2024-01-15T11:22:20.893 INFO:teuthology.orchestra.run.smithi113.stdout:(68/147): python3-webob-1.8.5-1.el8.1.noarch.rp 1.6 MB/s | 251 kB 00:00 2024-01-15T11:22:20.919 INFO:teuthology.orchestra.run.smithi113.stdout:(69/147): python3-webtest-2.0.33-1.el8.noarch.r 850 kB/s | 85 kB 00:00 2024-01-15T11:22:20.977 INFO:teuthology.orchestra.run.smithi113.stdout:(70/147): python3-zc-lockfile-2.0-2.el8.noarch. 228 kB/s | 23 kB 00:00 2024-01-15T11:22:21.036 INFO:teuthology.orchestra.run.smithi113.stdout:(71/147): xmlstarlet-1.6.1-20.el8.x86_64.rpm 595 kB/s | 70 kB 00:00 2024-01-15T11:22:21.120 INFO:teuthology.orchestra.run.smithi113.stdout:(72/147): lua-devel-5.3.4-12.el8.x86_64.rpm 211 kB/s | 30 kB 00:00 2024-01-15T11:22:21.145 INFO:teuthology.orchestra.run.smithi113.stdout:(73/147): libverto-devel-0.3.0-5.el8.x86_64.rpm 164 kB/s | 18 kB 00:00 2024-01-15T11:22:21.187 INFO:teuthology.orchestra.run.smithi113.stdout:(74/147): python3-jwt-1.6.1-2.el8.noarch.rpm 635 kB/s | 43 kB 00:00 2024-01-15T11:22:21.212 INFO:teuthology.orchestra.run.smithi113.stdout:(75/147): python3-oauthlib-2.1.0-1.el8.noarch.r 2.3 MB/s | 155 kB 00:00 2024-01-15T11:22:21.254 INFO:teuthology.orchestra.run.smithi113.stdout:(76/147): thrift-0.13.0-2.el8.x86_64.rpm 4.8 MB/s | 1.7 MB 00:00 2024-01-15T11:22:21.263 INFO:teuthology.orchestra.run.smithi141.stdout:(25/147): rbd-mirror-18.0.0-7629.g95c90f41.el8. 8.8 MB/s | 3.6 MB 00:00 2024-01-15T11:22:21.280 INFO:teuthology.orchestra.run.smithi113.stdout:(77/147): python3-requests-oauthlib-1.0.0-1.el8 636 kB/s | 43 kB 00:00 2024-01-15T11:22:21.313 INFO:teuthology.orchestra.run.smithi113.stdout:(78/147): python3-pycparser-2.14-14.el8.noarch. 867 kB/s | 109 kB 00:00 2024-01-15T11:22:21.322 INFO:teuthology.orchestra.run.smithi141.stdout:(26/147): rbd-nbd-18.0.0-7629.g95c90f41.el8.x86 3.0 MB/s | 181 kB 00:00 2024-01-15T11:22:21.355 INFO:teuthology.orchestra.run.smithi113.stdout:(79/147): libzstd-devel-1.4.4-1.el8.x86_64.rpm 627 kB/s | 44 kB 00:00 2024-01-15T11:22:21.372 INFO:teuthology.orchestra.run.smithi141.stdout:(27/147): ceph-grafana-dashboards-18.0.0-7629.g 561 kB/s | 28 kB 00:00 2024-01-15T11:22:21.381 INFO:teuthology.orchestra.run.smithi113.stdout:(80/147): libaio-devel-0.3.112-1.el8.x86_64.rpm 149 kB/s | 19 kB 00:00 2024-01-15T11:22:21.414 INFO:teuthology.orchestra.run.smithi113.stdout:(81/147): pcre2-devel-10.32-2.el8.x86_64.rpm 5.9 MB/s | 605 kB 00:00 2024-01-15T11:22:21.422 INFO:teuthology.orchestra.run.smithi141.stdout:(28/147): ceph-mgr-cephadm-18.0.0-7629.g95c90f4 2.6 MB/s | 133 kB 00:00 2024-01-15T11:22:21.439 INFO:teuthology.orchestra.run.smithi113.stdout:(82/147): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.7 MB/s | 229 kB 00:00 2024-01-15T11:22:21.465 INFO:teuthology.orchestra.run.smithi113.stdout:(83/147): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2024-01-15T11:22:21.514 INFO:teuthology.orchestra.run.smithi113.stdout:(84/147): keyutils-libs-devel-1.5.10-9.el8.x86_ 648 kB/s | 48 kB 00:00 2024-01-15T11:22:21.540 INFO:teuthology.orchestra.run.smithi113.stdout:(85/147): libselinux-devel-2.9-5.el8.x86_64.rpm 1.6 MB/s | 200 kB 00:00 2024-01-15T11:22:21.566 INFO:teuthology.orchestra.run.smithi113.stdout:(86/147): libsepol-devel-2.9-3.el8.x86_64.rpm 866 kB/s | 87 kB 00:00 2024-01-15T11:22:21.591 INFO:teuthology.orchestra.run.smithi113.stdout:(87/147): libcom_err-devel-1.45.6-4.el8.x86_64. 767 kB/s | 39 kB 00:00 2024-01-15T11:22:21.616 INFO:teuthology.orchestra.run.smithi113.stdout:(88/147): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 567 kB/s | 58 kB 00:00 2024-01-15T11:22:21.631 INFO:teuthology.orchestra.run.smithi141.stdout:(29/147): ceph-mgr-dashboard-18.0.0-7629.g95c90 8.4 MB/s | 1.8 MB 00:00 2024-01-15T11:22:21.641 INFO:teuthology.orchestra.run.smithi113.stdout:(89/147): fuse-2.9.7-17.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2024-01-15T11:22:21.667 INFO:teuthology.orchestra.run.smithi113.stdout:(90/147): libquadmath-8.5.0-20.el8.x86_64.rpm 3.4 MB/s | 172 kB 00:00 2024-01-15T11:22:21.692 INFO:teuthology.orchestra.run.smithi113.stdout:(91/147): libuuid-devel-2.32.1-43.el8.x86_64.rp 1.9 MB/s | 100 kB 00:00 2024-01-15T11:22:21.734 INFO:teuthology.orchestra.run.smithi113.stdout:(92/147): libgfortran-8.5.0-20.el8.x86_64.rpm 4.4 MB/s | 645 kB 00:00 2024-01-15T11:22:21.759 INFO:teuthology.orchestra.run.smithi113.stdout:(93/147): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.5 MB/s | 238 kB 00:00 2024-01-15T11:22:21.784 INFO:teuthology.orchestra.run.smithi113.stdout:(94/147): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 1.4 MB/s | 73 kB 00:00 2024-01-15T11:22:21.810 INFO:teuthology.orchestra.run.smithi113.stdout:(95/147): elfutils-libelf-devel-0.189-3.el8.x86 1.2 MB/s | 62 kB 00:00 2024-01-15T11:22:21.835 INFO:teuthology.orchestra.run.smithi113.stdout:(96/147): librabbitmq-0.9.0-4.el8.x86_64.rpm 934 kB/s | 47 kB 00:00 2024-01-15T11:22:21.869 INFO:teuthology.orchestra.run.smithi113.stdout:(97/147): python3-cryptography-3.2.1-6.el8.x86_ 3.1 MB/s | 559 kB 00:00 2024-01-15T11:22:21.902 INFO:teuthology.orchestra.run.smithi113.stdout:(98/147): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 6.0 MB/s | 562 kB 00:00 2024-01-15T11:22:21.993 INFO:teuthology.orchestra.run.smithi113.stdout:(99/147): openssl-devel-1.1.1k-12.el8_9.x86_64. 15 MB/s | 2.3 MB 00:00 2024-01-15T11:22:22.020 INFO:teuthology.orchestra.run.smithi113.stdout:(100/147): python3-jsonpointer-1.10-11.el8.noar 171 kB/s | 20 kB 00:00 2024-01-15T11:22:22.045 INFO:teuthology.orchestra.run.smithi113.stdout:(101/147): python3-prettytable-0.7.2-14.el8.noa 840 kB/s | 44 kB 00:00 2024-01-15T11:22:22.079 INFO:teuthology.orchestra.run.smithi113.stdout:(102/147): python3-pyasn1-modules-0.3.7-6.el8.n 1.8 MB/s | 110 kB 00:00 2024-01-15T11:22:22.104 INFO:teuthology.orchestra.run.smithi113.stdout:(103/147): python3-jsonpatch-1.21-2.el8.noarch. 453 kB/s | 27 kB 00:00 2024-01-15T11:22:22.129 INFO:teuthology.orchestra.run.smithi113.stdout:(104/147): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2024-01-15T11:22:22.163 INFO:teuthology.orchestra.run.smithi113.stdout:(105/147): python3-werkzeug-0.12.2-4.el8.noarch 7.6 MB/s | 457 kB 00:00 2024-01-15T11:22:22.205 INFO:teuthology.orchestra.run.smithi113.stdout:(106/147): bison-3.0.4-10.el8.x86_64.rpm 8.9 MB/s | 688 kB 00:00 2024-01-15T11:22:22.239 INFO:teuthology.orchestra.run.smithi113.stdout:(107/147): flex-2.6.1-9.el8.x86_64.rpm 4.2 MB/s | 320 kB 00:00 2024-01-15T11:22:22.272 INFO:teuthology.orchestra.run.smithi113.stdout:(108/147): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2024-01-15T11:22:22.289 INFO:teuthology.orchestra.run.smithi113.stdout:(109/147): python3-markupsafe-0.23-19.el8.x86_6 776 kB/s | 39 kB 00:00 2024-01-15T11:22:22.323 INFO:teuthology.orchestra.run.smithi113.stdout:(110/147): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.0 MB/s | 259 kB 00:00 2024-01-15T11:22:22.348 INFO:teuthology.orchestra.run.smithi113.stdout:(111/147): oniguruma-6.8.2-2.el8.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-01-15T11:22:22.356 INFO:teuthology.orchestra.run.smithi141.stdout:(30/147): ceph-test-18.0.0-7629.g95c90f41.el8.x 10 MB/s | 49 MB 00:04 2024-01-15T11:22:22.365 INFO:teuthology.orchestra.run.smithi113.stdout:(112/147): uuid-1.6.2-43.el8.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-01-15T11:22:22.398 INFO:teuthology.orchestra.run.smithi113.stdout:(113/147): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2024-01-15T11:22:22.415 INFO:teuthology.orchestra.run.smithi141.stdout:(31/147): ceph-mgr-modules-core-18.0.0-7629.g95 4.2 MB/s | 246 kB 00:00 2024-01-15T11:22:22.457 INFO:teuthology.orchestra.run.smithi141.stdout:(32/147): ceph-mgr-rook-18.0.0-7629.g95c90f41.e 1.2 MB/s | 50 kB 00:00 2024-01-15T11:22:22.487 INFO:teuthology.orchestra.run.smithi113.stdout:(114/147): python3-jinja2-2.10.1-3.el8.noarch.r 5.9 MB/s | 538 kB 00:00 2024-01-15T11:22:22.498 INFO:teuthology.orchestra.run.smithi141.stdout:(33/147): ceph-prometheus-alerts-18.0.0-7629.g9 362 kB/s | 15 kB 00:00 2024-01-15T11:22:22.540 INFO:teuthology.orchestra.run.smithi141.stdout:(34/147): ceph-volume-18.0.0-7629.g95c90f41.el8 6.1 MB/s | 262 kB 00:00 2024-01-15T11:22:22.591 INFO:teuthology.orchestra.run.smithi113.stdout:(115/147): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2024-01-15T11:22:22.641 INFO:teuthology.orchestra.run.smithi113.stdout:(116/147): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2024-01-15T11:22:22.708 INFO:teuthology.orchestra.run.smithi141.stdout:(35/147): ceph-mgr-diskprediction-local-18.0.0- 6.9 MB/s | 7.4 MB 00:01 2024-01-15T11:22:22.708 INFO:teuthology.orchestra.run.smithi113.stdout:(117/147): python3-numpy-1.14.3-10.el8.x86_64.r 17 MB/s | 3.7 MB 00:00 2024-01-15T11:22:22.733 INFO:teuthology.orchestra.run.smithi113.stdout:(118/147): NetworkManager-initscripts-updown-1. 167 kB/s | 144 kB 00:00 2024-01-15T11:22:22.758 INFO:teuthology.orchestra.run.smithi113.stdout:(119/147): python3-numpy-f2py-1.14.3-10.el8.x86 1.9 MB/s | 225 kB 00:00 2024-01-15T11:22:22.784 INFO:teuthology.orchestra.run.smithi113.stdout:(120/147): librdkafka-0.11.4-3.el8.x86_64.rpm 4.6 MB/s | 354 kB 00:00 2024-01-15T11:22:22.809 INFO:teuthology.orchestra.run.smithi113.stdout:(121/147): python3-mako-1.0.6-14.el8.noarch.rpm 3.0 MB/s | 157 kB 00:00 2024-01-15T11:22:22.885 INFO:teuthology.orchestra.run.smithi113.stdout:(122/147): automake-1.16.1-8.el8.noarch.rpm 9.3 MB/s | 714 kB 00:00 2024-01-15T11:22:22.916 INFO:teuthology.orchestra.run.smithi141.stdout:(36/147): ceph-common-18.0.0-7629.g95c90f41.el8 2.2 MB/s | 23 MB 00:10 2024-01-15T11:22:23.007 INFO:teuthology.orchestra.run.smithi113.stdout:(123/147): openblas-threads-0.3.15-6.el8.x86_64 19 MB/s | 5.2 MB 00:00 2024-01-15T11:22:23.008 INFO:teuthology.orchestra.run.smithi141.stdout:(37/147): cephfs-top-18.0.0-7629.g95c90f41.el8. 60 kB/s | 18 kB 00:00 2024-01-15T11:22:23.027 INFO:teuthology.orchestra.run.smithi113.stdout:(124/147): libpq-13.11-1.el8.x86_64.rpm 1.4 MB/s | 198 kB 00:00 2024-01-15T11:22:23.093 INFO:teuthology.orchestra.run.smithi113.stdout:(125/147): openblas-0.3.15-6.el8.x86_64.rpm 16 MB/s | 5.0 MB 00:00 2024-01-15T11:22:23.119 INFO:teuthology.orchestra.run.smithi113.stdout:(126/147): python3-pytz-2017.2-11.el8.noarch.rp 590 kB/s | 54 kB 00:00 2024-01-15T11:22:23.144 INFO:teuthology.orchestra.run.smithi113.stdout:(127/147): jq-1.6-7.el8.x86_64.rpm 1.4 MB/s | 202 kB 00:00 2024-01-15T11:22:23.194 INFO:teuthology.orchestra.run.smithi113.stdout:(128/147): postgresql-10.23-3.module+el8.9.0+20 15 MB/s | 1.5 MB 00:00 2024-01-15T11:22:23.242 INFO:teuthology.orchestra.run.smithi141.stdout:(38/147): cephadm-18.0.0-7629.g95c90f41.el8.noa 587 kB/s | 412 kB 00:00 2024-01-15T11:22:23.385 INFO:teuthology.orchestra.run.smithi141.stdout:(39/147): liboath-2.6.2-3.el8.x86_64.rpm 156 kB/s | 59 kB 00:00 2024-01-15T11:22:23.426 INFO:teuthology.orchestra.run.smithi141.stdout:(40/147): libunwind-1.3.1-3.el8.x86_64.rpm 429 kB/s | 75 kB 00:00 2024-01-15T11:22:23.484 INFO:teuthology.orchestra.run.smithi141.stdout:(41/147): gperftools-libs-2.7-9.el8.x86_64.rpm 539 kB/s | 306 kB 00:00 2024-01-15T11:22:23.509 INFO:teuthology.orchestra.run.smithi141.stdout:(42/147): luarocks-3.5.0-1.el8.noarch.rpm 1.3 MB/s | 167 kB 00:00 2024-01-15T11:22:23.543 INFO:teuthology.orchestra.run.smithi141.stdout:(43/147): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 747 kB/s | 44 kB 00:00 2024-01-15T11:22:23.544 INFO:teuthology.orchestra.run.smithi113.stdout:(129/147): postgresql-contrib-10.23-3.module+el 1.9 MB/s | 811 kB 00:00 2024-01-15T11:22:23.577 INFO:teuthology.orchestra.run.smithi141.stdout:(44/147): python3-asyncssh-2.7.0-2.el8.noarch.r 3.1 MB/s | 476 kB 00:00 2024-01-15T11:22:23.602 INFO:teuthology.orchestra.run.smithi141.stdout:(45/147): python3-cachetools-3.1.1-4.el8.noarch 564 kB/s | 33 kB 00:00 2024-01-15T11:22:23.627 INFO:teuthology.orchestra.run.smithi141.stdout:(46/147): python3-certifi-2018.10.15-7.el8.noar 326 kB/s | 16 kB 00:00 2024-01-15T11:22:23.644 INFO:teuthology.orchestra.run.smithi141.stdout:(47/147): python3-beautifulsoup4-4.6.3-2.el8.1. 1.3 MB/s | 185 kB 00:00 2024-01-15T11:22:23.661 INFO:teuthology.orchestra.run.smithi141.stdout:(48/147): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2024-01-15T11:22:23.662 INFO:teuthology.orchestra.run.smithi113.stdout:(130/147): postgresql-server-10.23-3.module+el8 9.8 MB/s | 5.1 MB 00:00 2024-01-15T11:22:23.721 INFO:teuthology.orchestra.run.smithi141.stdout:(49/147): python3-cherrypy-18.4.0-1.el8.noarch. 4.0 MB/s | 384 kB 00:00 2024-01-15T11:22:23.753 INFO:teuthology.orchestra.run.smithi141.stdout:(50/147): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2024-01-15T11:22:23.787 INFO:teuthology.orchestra.run.smithi141.stdout:(51/147): python3-jaraco-functools-2.0-4.el8.no 266 kB/s | 18 kB 00:00 2024-01-15T11:22:23.812 INFO:teuthology.orchestra.run.smithi141.stdout:(52/147): python3-google-auth-1.1.1-10.el8.noar 484 kB/s | 81 kB 00:00 2024-01-15T11:22:23.846 INFO:teuthology.orchestra.run.smithi141.stdout:(53/147): python3-logutils-0.3.5-11.el8.noarch. 836 kB/s | 49 kB 00:00 2024-01-15T11:22:23.871 INFO:teuthology.orchestra.run.smithi141.stdout:(54/147): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2024-01-15T11:22:23.887 INFO:teuthology.orchestra.run.smithi113.stdout:(131/147): python3-scipy-1.0.0-21.module+el8.9. 20 MB/s | 14 MB 00:00 2024-01-15T11:22:23.897 INFO:teuthology.orchestra.run.smithi141.stdout:(55/147): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2024-01-15T11:22:23.930 INFO:teuthology.orchestra.run.smithi141.stdout:(56/147): python3-kubernetes-11.0.0-6.el8.noarc 6.6 MB/s | 1.2 MB 00:00 2024-01-15T11:22:23.954 INFO:teuthology.orchestra.run.smithi113.stdout:(132/147): elfutils-libelf-0.189-3.el8.x86_64.r 3.4 MB/s | 232 kB 00:00 2024-01-15T11:22:23.956 INFO:teuthology.orchestra.run.smithi141.stdout:(57/147): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2024-01-15T11:22:23.981 INFO:teuthology.orchestra.run.smithi141.stdout:(58/147): python3-repoze-lru-0.7-6.el8.noarch.r 673 kB/s | 34 kB 00:00 2024-01-15T11:22:24.006 INFO:teuthology.orchestra.run.smithi141.stdout:(59/147): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2024-01-15T11:22:24.013 INFO:teuthology.orchestra.run.smithi113.stdout:(133/147): elfutils-libs-0.189-3.el8.x86_64.rpm 5.1 MB/s | 303 kB 00:00 2024-01-15T11:22:24.057 INFO:teuthology.orchestra.run.smithi141.stdout:(60/147): python3-pecan-1.3.2-9.el8.noarch.rpm 1.5 MB/s | 283 kB 00:00 2024-01-15T11:22:24.071 INFO:teuthology.orchestra.run.smithi113.stdout:(134/147): libmount-2.32.1-43.el8.x86_64.rpm 4.0 MB/s | 237 kB 00:00 2024-01-15T11:22:24.090 INFO:teuthology.orchestra.run.smithi141.stdout:(61/147): python3-simplegeneric-0.8.1-17.el8.no 224 kB/s | 19 kB 00:00 2024-01-15T11:22:24.107 INFO:teuthology.orchestra.run.smithi141.stdout:(62/147): python3-rsa-4.9-2.el8.noarch.rpm 511 kB/s | 64 kB 00:00 2024-01-15T11:22:24.121 INFO:teuthology.orchestra.run.smithi113.stdout:(135/147): libsmartcols-2.32.1-43.el8.x86_64.rp 3.5 MB/s | 179 kB 00:00 2024-01-15T11:22:24.132 INFO:teuthology.orchestra.run.smithi141.stdout:(63/147): python3-singledispatch-3.4.0.3-18.el8 320 kB/s | 24 kB 00:00 2024-01-15T11:22:24.157 INFO:teuthology.orchestra.run.smithi141.stdout:(64/147): python3-tempora-1.14.1-5.el8.noarch.r 437 kB/s | 29 kB 00:00 2024-01-15T11:22:24.172 INFO:teuthology.orchestra.run.smithi113.stdout:(136/147): libuuid-2.32.1-43.el8.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-01-15T11:22:24.182 INFO:teuthology.orchestra.run.smithi141.stdout:(65/147): python3-trustme-0.6.0-4.el8.noarch.rp 550 kB/s | 27 kB 00:00 2024-01-15T11:22:24.208 INFO:teuthology.orchestra.run.smithi141.stdout:(66/147): python3-waitress-1.4.3-1.el8.noarch.r 3.3 MB/s | 255 kB 00:00 2024-01-15T11:22:24.233 INFO:teuthology.orchestra.run.smithi141.stdout:(67/147): python3-webob-1.8.5-1.el8.1.noarch.rp 3.3 MB/s | 251 kB 00:00 2024-01-15T11:22:24.258 INFO:teuthology.orchestra.run.smithi141.stdout:(68/147): python3-websocket-client-0.56.0-5.el8 813 kB/s | 61 kB 00:00 2024-01-15T11:22:24.280 INFO:teuthology.orchestra.run.smithi113.stdout:(137/147): util-linux-2.32.1-43.el8.x86_64.rpm 23 MB/s | 2.5 MB 00:00 2024-01-15T11:22:24.283 INFO:teuthology.orchestra.run.smithi141.stdout:(69/147): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2024-01-15T11:22:24.309 INFO:teuthology.orchestra.run.smithi141.stdout:(70/147): python3-zc-lockfile-2.0-2.el8.noarch. 458 kB/s | 23 kB 00:00 2024-01-15T11:22:24.331 INFO:teuthology.orchestra.run.smithi113.stdout:(138/147): util-linux-user-2.32.1-43.el8.x86_64 2.0 MB/s | 103 kB 00:00 2024-01-15T11:22:24.334 INFO:teuthology.orchestra.run.smithi141.stdout:(71/147): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2024-01-15T11:22:24.401 INFO:teuthology.orchestra.run.smithi141.stdout:(72/147): libverto-devel-0.3.0-5.el8.x86_64.rpm 195 kB/s | 18 kB 00:00 2024-01-15T11:22:24.406 INFO:teuthology.orchestra.run.smithi113.stdout:(139/147): iproute-6.2.0-5.el8_9.x86_64.rpm 11 MB/s | 854 kB 00:00 2024-01-15T11:22:24.426 INFO:teuthology.orchestra.run.smithi141.stdout:(73/147): python3-jwt-1.6.1-2.el8.noarch.rpm 494 kB/s | 43 kB 00:00 2024-01-15T11:22:24.468 INFO:teuthology.orchestra.run.smithi141.stdout:(74/147): thrift-0.13.0-2.el8.x86_64.rpm 8.3 MB/s | 1.7 MB 00:00 2024-01-15T11:22:24.484 INFO:teuthology.orchestra.run.smithi113.stdout:(140/147): librados2-18.0.0-7629.g95c90f41.el8. 4.0 MB/s | 3.8 MB 00:00 2024-01-15T11:22:24.494 INFO:teuthology.orchestra.run.smithi141.stdout:(75/147): python3-oauthlib-2.1.0-1.el8.noarch.r 1.6 MB/s | 155 kB 00:00 2024-01-15T11:22:24.515 INFO:teuthology.orchestra.run.smithi113.stdout:(141/147): libblkid-2.32.1-43.el8.x86_64.rpm 2.0 MB/s | 220 kB 00:00 2024-01-15T11:22:24.519 INFO:teuthology.orchestra.run.smithi141.stdout:(76/147): python3-requests-oauthlib-1.0.0-1.el8 849 kB/s | 43 kB 00:00 2024-01-15T11:22:24.544 INFO:teuthology.orchestra.run.smithi141.stdout:(77/147): python3-pycparser-2.14-14.el8.noarch. 929 kB/s | 109 kB 00:00 2024-01-15T11:22:24.549 INFO:teuthology.orchestra.run.smithi113.stdout:(142/147): libfdisk-2.32.1-43.el8.x86_64.rpm 3.9 MB/s | 254 kB 00:00 2024-01-15T11:22:24.561 INFO:teuthology.orchestra.run.smithi141.stdout:(78/147): libaio-devel-0.3.112-1.el8.x86_64.rpm 278 kB/s | 19 kB 00:00 2024-01-15T11:22:24.578 INFO:teuthology.orchestra.run.smithi141.stdout:(79/147): libzstd-devel-1.4.4-1.el8.x86_64.rpm 824 kB/s | 44 kB 00:00 2024-01-15T11:22:24.590 INFO:teuthology.orchestra.run.smithi113.stdout:(143/147): xfsprogs-5.0.0-12.el8.x86_64.rpm 14 MB/s | 1.1 MB 00:00 2024-01-15T11:22:24.603 INFO:teuthology.orchestra.run.smithi141.stdout:(80/147): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.4 MB/s | 229 kB 00:00 2024-01-15T11:22:24.624 INFO:teuthology.orchestra.run.smithi113.stdout:(144/147): openssl-1.1.1k-12.el8_9.x86_64.rpm 9.2 MB/s | 711 kB 00:00 2024-01-15T11:22:24.629 INFO:teuthology.orchestra.run.smithi141.stdout:(81/147): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-01-15T11:22:24.655 INFO:teuthology.orchestra.run.smithi141.stdout:(82/147): pcre2-devel-10.32-2.el8.x86_64.rpm 5.4 MB/s | 605 kB 00:00 2024-01-15T11:22:24.671 INFO:teuthology.orchestra.run.smithi141.stdout:(83/147): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.1 MB/s | 48 kB 00:00 2024-01-15T11:22:24.674 INFO:teuthology.orchestra.run.smithi113.stdout:(145/147): openssl-libs-1.1.1k-12.el8_9.x86_64. 18 MB/s | 1.5 MB 00:00 2024-01-15T11:22:24.721 INFO:teuthology.orchestra.run.smithi141.stdout:(84/147): libselinux-devel-2.9-5.el8.x86_64.rpm 1.7 MB/s | 200 kB 00:00 2024-01-15T11:22:24.733 INFO:teuthology.orchestra.run.smithi113.stdout:(146/147): librbd1-18.0.0-7629.g95c90f41.el8.x8 3.5 MB/s | 3.7 MB 00:01 2024-01-15T11:22:24.747 INFO:teuthology.orchestra.run.smithi141.stdout:(85/147): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 769 kB/s | 58 kB 00:00 2024-01-15T11:22:24.764 INFO:teuthology.orchestra.run.smithi141.stdout:(86/147): libsepol-devel-2.9-3.el8.x86_64.rpm 798 kB/s | 87 kB 00:00 2024-01-15T11:22:24.789 INFO:teuthology.orchestra.run.smithi141.stdout:(87/147): libcom_err-devel-1.45.6-4.el8.x86_64. 567 kB/s | 39 kB 00:00 2024-01-15T11:22:24.814 INFO:teuthology.orchestra.run.smithi141.stdout:(88/147): fuse-2.9.7-17.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2024-01-15T11:22:24.839 INFO:teuthology.orchestra.run.smithi141.stdout:(89/147): libquadmath-8.5.0-20.el8.x86_64.rpm 3.4 MB/s | 172 kB 00:00 2024-01-15T11:22:24.865 INFO:teuthology.orchestra.run.smithi141.stdout:(90/147): libuuid-devel-2.32.1-43.el8.x86_64.rp 1.9 MB/s | 100 kB 00:00 2024-01-15T11:22:24.890 INFO:teuthology.orchestra.run.smithi141.stdout:(91/147): python3-cffi-1.11.5-6.el8.x86_64.rpm 4.6 MB/s | 238 kB 00:00 2024-01-15T11:22:24.915 INFO:teuthology.orchestra.run.smithi141.stdout:(92/147): libgfortran-8.5.0-20.el8.x86_64.rpm 4.2 MB/s | 645 kB 00:00 2024-01-15T11:22:24.932 INFO:teuthology.orchestra.run.smithi141.stdout:(93/147): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 1.7 MB/s | 73 kB 00:00 2024-01-15T11:22:24.957 INFO:teuthology.orchestra.run.smithi141.stdout:(94/147): elfutils-libelf-devel-0.189-3.el8.x86 1.4 MB/s | 62 kB 00:00 2024-01-15T11:22:24.983 INFO:teuthology.orchestra.run.smithi141.stdout:(95/147): librabbitmq-0.9.0-4.el8.x86_64.rpm 936 kB/s | 47 kB 00:00 2024-01-15T11:22:25.017 INFO:teuthology.orchestra.run.smithi141.stdout:(96/147): python3-cryptography-3.2.1-6.el8.x86_ 3.6 MB/s | 559 kB 00:00 2024-01-15T11:22:25.067 INFO:teuthology.orchestra.run.smithi141.stdout:(97/147): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 5.0 MB/s | 562 kB 00:00 2024-01-15T11:22:25.092 INFO:teuthology.orchestra.run.smithi141.stdout:(98/147): NetworkManager-initscripts-updown-1.4 1.9 MB/s | 144 kB 00:00 2024-01-15T11:22:25.117 INFO:teuthology.orchestra.run.smithi141.stdout:(99/147): lua-devel-5.3.4-12.el8.x86_64.rpm 598 kB/s | 30 kB 00:00 2024-01-15T11:22:25.142 INFO:teuthology.orchestra.run.smithi141.stdout:(100/147): python3-jsonpointer-1.10-11.el8.noar 399 kB/s | 20 kB 00:00 2024-01-15T11:22:25.168 INFO:teuthology.orchestra.run.smithi141.stdout:(101/147): python3-prettytable-0.7.2-14.el8.noa 878 kB/s | 44 kB 00:00 2024-01-15T11:22:25.193 INFO:teuthology.orchestra.run.smithi141.stdout:(102/147): python3-pyasn1-modules-0.3.7-6.el8.n 2.1 MB/s | 110 kB 00:00 2024-01-15T11:22:25.218 INFO:teuthology.orchestra.run.smithi141.stdout:(103/147): python3-jsonpatch-1.21-2.el8.noarch. 530 kB/s | 27 kB 00:00 2024-01-15T11:22:25.244 INFO:teuthology.orchestra.run.smithi141.stdout:(104/147): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2024-01-15T11:22:25.277 INFO:teuthology.orchestra.run.smithi141.stdout:(105/147): python3-werkzeug-0.12.2-4.el8.noarch 7.6 MB/s | 457 kB 00:00 2024-01-15T11:22:25.311 INFO:teuthology.orchestra.run.smithi141.stdout:(106/147): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2024-01-15T11:22:25.344 INFO:teuthology.orchestra.run.smithi141.stdout:(107/147): flex-2.6.1-9.el8.x86_64.rpm 4.7 MB/s | 320 kB 00:00 2024-01-15T11:22:25.411 INFO:teuthology.orchestra.run.smithi141.stdout:(108/147): openssl-devel-1.1.1k-12.el8_9.x86_64 5.4 MB/s | 2.3 MB 00:00 2024-01-15T11:22:25.437 INFO:teuthology.orchestra.run.smithi141.stdout:(109/147): python3-markupsafe-0.23-19.el8.x86_6 422 kB/s | 39 kB 00:00 2024-01-15T11:22:25.462 INFO:teuthology.orchestra.run.smithi141.stdout:(110/147): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.0 MB/s | 259 kB 00:00 2024-01-15T11:22:25.487 INFO:teuthology.orchestra.run.smithi141.stdout:(111/147): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-01-15T11:22:25.513 INFO:teuthology.orchestra.run.smithi141.stdout:(112/147): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-01-15T11:22:25.525 INFO:teuthology.orchestra.run.smithi113.stdout:(147/147): xfsdump-3.1.8-7.el8_9.x86_64.rpm 370 kB/s | 333 kB 00:00 2024-01-15T11:22:25.528 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-01-15T11:22:25.528 INFO:teuthology.orchestra.run.smithi113.stdout:Total 15 MB/s | 218 MB 00:14 2024-01-15T11:22:25.538 INFO:teuthology.orchestra.run.smithi141.stdout:(113/147): libtool-2.4.6-25.el8.x86_64.rpm 3.0 MB/s | 709 kB 00:00 2024-01-15T11:22:25.563 INFO:teuthology.orchestra.run.smithi141.stdout:(114/147): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.3 MB/s | 103 kB 00:00 2024-01-15T11:22:25.597 INFO:teuthology.orchestra.run.smithi141.stdout:(115/147): python3-jinja2-2.10.1-3.el8.noarch.r 9.0 MB/s | 538 kB 00:00 2024-01-15T11:22:25.639 INFO:teuthology.orchestra.run.smithi141.stdout:(116/147): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2024-01-15T11:22:25.706 INFO:teuthology.orchestra.run.smithi141.stdout:(117/147): python3-numpy-f2py-1.14.3-10.el8.x86 3.3 MB/s | 225 kB 00:00 2024-01-15T11:22:25.756 INFO:teuthology.orchestra.run.smithi141.stdout:(118/147): librdkafka-0.11.4-3.el8.x86_64.rpm 7.0 MB/s | 354 kB 00:00 2024-01-15T11:22:25.815 INFO:teuthology.orchestra.run.smithi141.stdout:(119/147): python3-numpy-1.14.3-10.el8.x86_64.r 15 MB/s | 3.7 MB 00:00 2024-01-15T11:22:25.882 INFO:teuthology.orchestra.run.smithi141.stdout:(120/147): python3-babel-2.5.1-7.el8.noarch.rpm 13 MB/s | 4.8 MB 00:00 2024-01-15T11:22:25.899 INFO:teuthology.orchestra.run.smithi141.stdout:(121/147): python3-mako-1.0.6-14.el8.noarch.rpm 1.8 MB/s | 157 kB 00:00 2024-01-15T11:22:25.957 INFO:teuthology.orchestra.run.smithi141.stdout:(122/147): automake-1.16.1-8.el8.noarch.rpm 12 MB/s | 714 kB 00:00 2024-01-15T11:22:25.991 INFO:teuthology.orchestra.run.smithi141.stdout:(123/147): libpq-13.11-1.el8.x86_64.rpm 5.8 MB/s | 198 kB 00:00 2024-01-15T11:22:26.049 INFO:teuthology.orchestra.run.smithi141.stdout:(124/147): jq-1.6-7.el8.x86_64.rpm 3.4 MB/s | 202 kB 00:00 2024-01-15T11:22:26.083 INFO:teuthology.orchestra.run.smithi141.stdout:(125/147): python3-pytz-2017.2-11.el8.noarch.rp 1.6 MB/s | 54 kB 00:00 2024-01-15T11:22:26.150 INFO:teuthology.orchestra.run.smithi141.stdout:(126/147): openblas-threads-0.3.15-6.el8.x86_64 13 MB/s | 5.2 MB 00:00 2024-01-15T11:22:26.217 INFO:teuthology.orchestra.run.smithi141.stdout:(127/147): openblas-0.3.15-6.el8.x86_64.rpm 15 MB/s | 5.0 MB 00:00 2024-01-15T11:22:26.259 INFO:teuthology.orchestra.run.smithi141.stdout:(128/147): postgresql-contrib-10.23-3.module+el 7.3 MB/s | 811 kB 00:00 2024-01-15T11:22:26.293 INFO:teuthology.orchestra.run.smithi141.stdout:(129/147): postgresql-10.23-3.module+el8.9.0+20 7.2 MB/s | 1.5 MB 00:00 2024-01-15T11:22:26.418 INFO:teuthology.orchestra.run.smithi141.stdout:(130/147): postgresql-server-10.23-3.module+el8 25 MB/s | 5.1 MB 00:00 2024-01-15T11:22:26.594 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-01-15T11:22:26.643 INFO:teuthology.orchestra.run.smithi141.stdout:(131/147): python3-scipy-1.0.0-21.module+el8.9. 36 MB/s | 14 MB 00:00 2024-01-15T11:22:26.710 INFO:teuthology.orchestra.run.smithi141.stdout:(132/147): elfutils-libelf-0.189-3.el8.x86_64.r 3.4 MB/s | 232 kB 00:00 2024-01-15T11:22:26.743 INFO:teuthology.orchestra.run.smithi141.stdout:(133/147): elfutils-libs-0.189-3.el8.x86_64.rpm 8.9 MB/s | 303 kB 00:00 2024-01-15T11:22:26.786 INFO:teuthology.orchestra.run.smithi141.stdout:(134/147): libmount-2.32.1-43.el8.x86_64.rpm 5.6 MB/s | 237 kB 00:00 2024-01-15T11:22:26.819 INFO:teuthology.orchestra.run.smithi141.stdout:(135/147): libsmartcols-2.32.1-43.el8.x86_64.rp 5.3 MB/s | 179 kB 00:00 2024-01-15T11:22:26.852 INFO:teuthology.orchestra.run.smithi141.stdout:(136/147): libuuid-2.32.1-43.el8.x86_64.rpm 2.9 MB/s | 99 kB 00:00 2024-01-15T11:22:26.936 INFO:teuthology.orchestra.run.smithi141.stdout:(137/147): librados2-18.0.0-7629.g95c90f41.el8. 5.9 MB/s | 3.8 MB 00:00 2024-01-15T11:22:26.994 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-01-15T11:22:26.994 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-01-15T11:22:27.013 INFO:teuthology.orchestra.run.smithi141.stdout:(138/147): librbd1-18.0.0-7629.g95c90f41.el8.x8 6.2 MB/s | 3.7 MB 00:00 2024-01-15T11:22:27.045 INFO:teuthology.orchestra.run.smithi141.stdout:(139/147): util-linux-user-2.32.1-43.el8.x86_64 945 kB/s | 103 kB 00:00 2024-01-15T11:22:27.096 INFO:teuthology.orchestra.run.smithi141.stdout:(140/147): iproute-6.2.0-5.el8_9.x86_64.rpm 10 MB/s | 854 kB 00:00 2024-01-15T11:22:27.121 INFO:teuthology.orchestra.run.smithi141.stdout:(141/147): libblkid-2.32.1-43.el8.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-01-15T11:22:27.171 INFO:teuthology.orchestra.run.smithi141.stdout:(142/147): util-linux-2.32.1-43.el8.x86_64.rpm 7.8 MB/s | 2.5 MB 00:00 2024-01-15T11:22:27.196 INFO:teuthology.orchestra.run.smithi141.stdout:(143/147): libfdisk-2.32.1-43.el8.x86_64.rpm 2.5 MB/s | 254 kB 00:00 2024-01-15T11:22:27.230 INFO:teuthology.orchestra.run.smithi141.stdout:(144/147): openssl-1.1.1k-12.el8_9.x86_64.rpm 12 MB/s | 711 kB 00:00 2024-01-15T11:22:27.264 INFO:teuthology.orchestra.run.smithi141.stdout:(145/147): xfsprogs-5.0.0-12.el8.x86_64.rpm 7.5 MB/s | 1.1 MB 00:00 2024-01-15T11:22:27.297 INFO:teuthology.orchestra.run.smithi141.stdout:(146/147): xfsdump-3.1.8-7.el8_9.x86_64.rpm 4.9 MB/s | 333 kB 00:00 2024-01-15T11:22:27.364 INFO:teuthology.orchestra.run.smithi141.stdout:(147/147): openssl-libs-1.1.1k-12.el8_9.x86_64. 8.8 MB/s | 1.5 MB 00:00 2024-01-15T11:22:27.368 INFO:teuthology.orchestra.run.smithi141.stdout:-------------------------------------------------------------------------------- 2024-01-15T11:22:27.368 INFO:teuthology.orchestra.run.smithi141.stdout:Total 15 MB/s | 218 MB 00:14 2024-01-15T11:22:28.208 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-01-15T11:22:28.218 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-01-15T11:22:28.442 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction check 2024-01-15T11:22:28.839 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction check succeeded. 2024-01-15T11:22:28.839 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction test 2024-01-15T11:22:30.062 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction test succeeded. 2024-01-15T11:22:30.072 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction 2024-01-15T11:22:33.905 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-01-15T11:22:34.149 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-15T11:22:34.493 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/163 2024-01-15T11:22:34.716 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/163 2024-01-15T11:22:34.889 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/163 2024-01-15T11:22:35.043 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/163 2024-01-15T11:22:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/163 2024-01-15T11:22:35.318 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/163 2024-01-15T11:22:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/163 2024-01-15T11:22:35.564 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/163 2024-01-15T11:22:35.791 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 6/163 2024-01-15T11:22:35.950 INFO:teuthology.orchestra.run.smithi141.stdout: Preparing : 1/1 2024-01-15T11:22:35.984 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/163 2024-01-15T11:22:36.085 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-15T11:22:36.087 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/163 2024-01-15T11:22:36.293 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/163 2024-01-15T11:22:36.353 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/163 2024-01-15T11:22:36.571 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/163 2024-01-15T11:22:36.707 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/163 2024-01-15T11:22:36.721 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/163 2024-01-15T11:22:36.856 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/163 2024-01-15T11:22:36.909 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 9/163 2024-01-15T11:22:36.965 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/163 2024-01-15T11:22:37.156 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/163 2024-01-15T11:22:37.259 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/163 2024-01-15T11:22:37.315 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 9/163 2024-01-15T11:22:37.452 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/163 2024-01-15T11:22:37.493 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 10/163 2024-01-15T11:22:37.623 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 10/163 2024-01-15T11:22:37.671 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 6/163 2024-01-15T11:22:37.793 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 11/163 2024-01-15T11:22:37.831 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/163 2024-01-15T11:22:37.874 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 11/163 2024-01-15T11:22:37.941 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/163 2024-01-15T11:22:38.123 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/163 2024-01-15T11:22:38.528 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/163 2024-01-15T11:22:38.647 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rados-2:18.0.0-7629.g95c90f41.el8.x86_64 12/163 2024-01-15T11:22:38.773 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 9/163 2024-01-15T11:22:38.800 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 13/163 2024-01-15T11:22:38.982 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 13/163 2024-01-15T11:22:39.209 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/163 2024-01-15T11:22:39.298 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/163 2024-01-15T11:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libpq-13.11-1.el8.x86_64 15/163 2024-01-15T11:22:39.550 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 9/163 2024-01-15T11:22:39.748 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 10/163 2024-01-15T11:22:39.880 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 10/163 2024-01-15T11:22:39.981 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 16/163 2024-01-15T11:22:40.081 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 11/163 2024-01-15T11:22:40.180 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 11/163 2024-01-15T11:22:40.196 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/163 2024-01-15T11:22:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 18/163 2024-01-15T11:22:40.774 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 19/163 2024-01-15T11:22:40.908 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 20/163 2024-01-15T11:22:40.980 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-rados-2:18.0.0-7629.g95c90f41.el8.x86_64 12/163 2024-01-15T11:22:41.001 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 20/163 2024-01-15T11:22:41.114 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 13/163 2024-01-15T11:22:41.167 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86 21/163 2024-01-15T11:22:41.246 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86 21/163 2024-01-15T11:22:41.289 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 13/163 2024-01-15T11:22:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 22/163 2024-01-15T11:22:41.547 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/163 2024-01-15T11:22:41.655 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/163 2024-01-15T11:22:41.684 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/163 2024-01-15T11:22:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libpq-13.11-1.el8.x86_64 15/163 2024-01-15T11:22:41.938 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/163 2024-01-15T11:22:42.050 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 16/163 2024-01-15T11:22:42.133 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 25/163 2024-01-15T11:22:42.234 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/163 2024-01-15T11:22:42.256 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 25/163 2024-01-15T11:22:42.400 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 26/163 2024-01-15T11:22:42.626 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 18/163 2024-01-15T11:22:42.837 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 19/163 2024-01-15T11:22:43.013 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 20/163 2024-01-15T11:22:43.023 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 26/163 2024-01-15T11:22:43.124 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 20/163 2024-01-15T11:22:43.175 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 27/163 2024-01-15T11:22:43.314 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86 21/163 2024-01-15T11:22:43.401 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86 21/163 2024-01-15T11:22:43.650 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 22/163 2024-01-15T11:22:43.763 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 27/163 2024-01-15T11:22:43.931 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/163 2024-01-15T11:22:43.958 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-0.3.15-6.el8.x86_64 28/163 2024-01-15T11:22:44.177 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/163 2024-01-15T11:22:44.363 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 25/163 2024-01-15T11:22:44.481 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 25/163 2024-01-15T11:22:44.623 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 28/163 2024-01-15T11:22:44.663 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 26/163 2024-01-15T11:22:44.880 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 29/163 2024-01-15T11:22:45.113 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 30/163 2024-01-15T11:22:45.297 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 26/163 2024-01-15T11:22:45.376 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 31/163 2024-01-15T11:22:45.488 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 27/163 2024-01-15T11:22:45.556 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g95c90f41.el8. 32/163 2024-01-15T11:22:45.713 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el 33/163 2024-01-15T11:22:46.020 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cephfs-2:18.0.0-7629.g95c90f41.el8.x86_6 34/163 2024-01-15T11:22:46.095 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 27/163 2024-01-15T11:22:46.199 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/163 2024-01-15T11:22:46.313 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : openblas-0.3.15-6.el8.x86_64 28/163 2024-01-15T11:22:46.998 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 28/163 2024-01-15T11:22:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 29/163 2024-01-15T11:22:47.476 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 30/163 2024-01-15T11:22:47.739 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 31/163 2024-01-15T11:22:47.902 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g95c90f41.el8. 32/163 2024-01-15T11:22:47.926 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/163 2024-01-15T11:22:48.110 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el 33/163 2024-01-15T11:22:48.292 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 36/163 2024-01-15T11:22:48.417 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cephfs-2:18.0.0-7629.g95c90f41.el8.x86_6 34/163 2024-01-15T11:22:48.654 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/163 2024-01-15T11:22:49.286 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 37/163 2024-01-15T11:22:49.619 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 38/163 2024-01-15T11:22:49.857 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 39/163 2024-01-15T11:22:50.075 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rsa-4.9-2.el8.noarch 40/163 2024-01-15T11:22:50.268 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 41/163 2024-01-15T11:22:50.394 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/163 2024-01-15T11:22:50.492 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rgw-2:18.0.0-7629.g95c90f41.el8.x86_64 42/163 2024-01-15T11:22:50.704 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rbd-2:18.0.0-7629.g95c90f41.el8.x86_64 43/163 2024-01-15T11:22:50.765 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 36/163 2024-01-15T11:22:50.937 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librados-devel-2:18.0.0-7629.g95c90f41.el8.x86_6 44/163 2024-01-15T11:22:51.165 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 45/163 2024-01-15T11:22:51.266 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 45/163 2024-01-15T11:22:51.489 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 46/163 2024-01-15T11:22:51.559 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 46/163 2024-01-15T11:22:51.760 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 37/163 2024-01-15T11:22:51.786 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 47/163 2024-01-15T11:22:52.005 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 48/163 2024-01-15T11:22:52.125 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 38/163 2024-01-15T11:22:52.270 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 49/163 2024-01-15T11:22:52.379 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 39/163 2024-01-15T11:22:52.606 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-rsa-4.9-2.el8.noarch 40/163 2024-01-15T11:22:52.695 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 49/163 2024-01-15T11:22:52.781 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 41/163 2024-01-15T11:22:53.006 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 50/163 2024-01-15T11:22:53.030 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-rgw-2:18.0.0-7629.g95c90f41.el8.x86_64 42/163 2024-01-15T11:22:53.097 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 50/163 2024-01-15T11:22:53.250 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-rbd-2:18.0.0-7629.g95c90f41.el8.x86_64 43/163 2024-01-15T11:22:53.331 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : automake-1.16.1-8.el8.noarch 51/163 2024-01-15T11:22:53.475 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : librados-devel-2:18.0.0-7629.g95c90f41.el8.x86_6 44/163 2024-01-15T11:22:53.506 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : uuid-1.6.2-43.el8.x86_64 52/163 2024-01-15T11:22:53.584 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 52/163 2024-01-15T11:22:53.703 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 45/163 2024-01-15T11:22:53.772 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 53/163 2024-01-15T11:22:53.779 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 45/163 2024-01-15T11:22:53.850 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 53/163 2024-01-15T11:22:53.995 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 46/163 2024-01-15T11:22:54.055 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 46/163 2024-01-15T11:22:54.055 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : jq-1.6-7.el8.x86_64 54/163 2024-01-15T11:22:54.229 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 55/163 2024-01-15T11:22:54.302 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 47/163 2024-01-15T11:22:54.405 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 56/163 2024-01-15T11:22:54.525 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 48/163 2024-01-15T11:22:54.615 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 57/163 2024-01-15T11:22:54.766 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 49/163 2024-01-15T11:22:54.843 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : fuse-2.9.7-17.el8.x86_64 58/163 2024-01-15T11:22:55.054 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 59/163 2024-01-15T11:22:55.194 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 49/163 2024-01-15T11:22:55.253 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 60/163 2024-01-15T11:22:55.434 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 61/163 2024-01-15T11:22:55.545 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 50/163 2024-01-15T11:22:55.639 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 50/163 2024-01-15T11:22:55.668 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 62/163 2024-01-15T11:22:55.854 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : automake-1.16.1-8.el8.noarch 51/163 2024-01-15T11:22:55.966 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 63/163 2024-01-15T11:22:56.053 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : uuid-1.6.2-43.el8.x86_64 52/163 2024-01-15T11:22:56.139 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 52/163 2024-01-15T11:22:56.260 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/163 2024-01-15T11:22:56.353 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 53/163 2024-01-15T11:22:56.429 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 53/163 2024-01-15T11:22:56.477 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/163 2024-01-15T11:22:56.644 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : jq-1.6-7.el8.x86_64 54/163 2024-01-15T11:22:56.676 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 66/163 2024-01-15T11:22:56.809 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 55/163 2024-01-15T11:22:56.883 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 67/163 2024-01-15T11:22:57.027 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 56/163 2024-01-15T11:22:57.207 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 57/163 2024-01-15T11:22:57.280 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 68/163 2024-01-15T11:22:57.396 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : lua-devel-5.3.4-12.el8.x86_64 58/163 2024-01-15T11:22:57.573 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 69/163 2024-01-15T11:22:57.648 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : fuse-2.9.7-17.el8.x86_64 59/163 2024-01-15T11:22:57.775 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 70/163 2024-01-15T11:22:57.859 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 60/163 2024-01-15T11:22:58.000 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 71/163 2024-01-15T11:22:58.100 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/163 2024-01-15T11:22:58.242 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 72/163 2024-01-15T11:22:58.307 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/163 2024-01-15T11:22:58.490 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/163 2024-01-15T11:22:58.526 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 73/163 2024-01-15T11:22:58.729 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 74/163 2024-01-15T11:22:58.815 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 64/163 2024-01-15T11:22:58.888 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 75/163 2024-01-15T11:22:59.132 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 76/163 2024-01-15T11:22:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 65/163 2024-01-15T11:22:59.326 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 77/163 2024-01-15T11:22:59.366 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 66/163 2024-01-15T11:22:59.523 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 67/163 2024-01-15T11:22:59.626 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 78/163 2024-01-15T11:22:59.762 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 68/163 2024-01-15T11:22:59.781 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lua-devel-5.3.4-12.el8.x86_64 79/163 2024-01-15T11:22:59.988 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 80/163 2024-01-15T11:23:00.167 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 81/163 2024-01-15T11:23:00.201 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 69/163 2024-01-15T11:23:00.417 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 82/163 2024-01-15T11:23:00.503 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 70/163 2024-01-15T11:23:00.678 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 83/163 2024-01-15T11:23:00.697 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 71/163 2024-01-15T11:23:00.851 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 84/163 2024-01-15T11:23:00.938 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 72/163 2024-01-15T11:23:01.066 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 85/163 2024-01-15T11:23:01.156 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 73/163 2024-01-15T11:23:01.288 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 86/163 2024-01-15T11:23:01.500 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/163 2024-01-15T11:23:01.533 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 87/163 2024-01-15T11:23:01.735 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 75/163 2024-01-15T11:23:01.773 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 88/163 2024-01-15T11:23:01.944 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 76/163 2024-01-15T11:23:01.991 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 89/163 2024-01-15T11:23:02.146 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 77/163 2024-01-15T11:23:02.193 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 90/163 2024-01-15T11:23:02.391 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 78/163 2024-01-15T11:23:02.473 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 91/163 2024-01-15T11:23:02.670 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 79/163 2024-01-15T11:23:02.711 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 92/163 2024-01-15T11:23:02.818 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 80/163 2024-01-15T11:23:02.901 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 93/163 2024-01-15T11:23:03.030 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 81/163 2024-01-15T11:23:03.150 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-portend-2.6-1.el8.noarch 94/163 2024-01-15T11:23:03.287 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 82/163 2024-01-15T11:23:03.419 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 95/163 2024-01-15T11:23:03.491 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 83/163 2024-01-15T11:23:03.594 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 96/163 2024-01-15T11:23:03.723 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 84/163 2024-01-15T11:23:03.811 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 97/163 2024-01-15T11:23:03.946 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 85/163 2024-01-15T11:23:04.170 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 86/163 2024-01-15T11:23:04.422 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 87/163 2024-01-15T11:23:04.669 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 88/163 2024-01-15T11:23:04.888 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 89/163 2024-01-15T11:23:04.910 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 98/163 2024-01-15T11:23:05.206 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 99/163 2024-01-15T11:23:05.427 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 100/163 2024-01-15T11:23:05.499 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 90/163 2024-01-15T11:23:05.702 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 101/163 2024-01-15T11:23:05.772 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 91/163 2024-01-15T11:23:06.039 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 92/163 2024-01-15T11:23:06.105 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 102/163 2024-01-15T11:23:06.207 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 93/163 2024-01-15T11:23:06.388 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : luarocks-3.5.0-1.el8.noarch 103/163 2024-01-15T11:23:06.506 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-portend-2.6-1.el8.noarch 94/163 2024-01-15T11:23:06.586 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 104/163 2024-01-15T11:23:06.759 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 95/163 2024-01-15T11:23:06.779 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 105/163 2024-01-15T11:23:06.983 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 96/163 2024-01-15T11:23:07.201 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 97/163 2024-01-15T11:23:08.285 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 98/163 2024-01-15T11:23:08.621 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 99/163 2024-01-15T11:23:08.841 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 100/163 2024-01-15T11:23:09.209 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 101/163 2024-01-15T11:23:09.528 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 102/163 2024-01-15T11:23:09.785 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : luarocks-3.5.0-1.el8.noarch 103/163 2024-01-15T11:23:10.009 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 104/163 2024-01-15T11:23:10.226 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 105/163 2024-01-15T11:23:10.494 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 106/163 2024-01-15T11:23:10.736 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 106/163 2024-01-15T11:23:11.333 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 106/163 2024-01-15T11:23:12.178 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 107/163 2024-01-15T11:23:12.267 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 107/163 2024-01-15T11:23:12.452 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 108/163 2024-01-15T11:23:14.482 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 106/163 2024-01-15T11:23:14.742 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 106/163 2024-01-15T11:23:15.316 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 106/163 2024-01-15T11:23:15.588 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 107/163 2024-01-15T11:23:15.681 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 107/163 2024-01-15T11:23:15.841 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 108/163 2024-01-15T11:23:31.791 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 108/163 2024-01-15T11:23:31.791 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /sys 2024-01-15T11:23:31.791 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /proc 2024-01-15T11:23:31.791 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /dev 2024-01-15T11:23:31.791 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /run 2024-01-15T11:23:31.791 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /mnt 2024-01-15T11:23:31.791 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /var/tmp 2024-01-15T11:23:31.791 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /home 2024-01-15T11:23:31.791 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /tmp 2024-01-15T11:23:31.791 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /dev 2024-01-15T11:23:31.792 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:23:32.182 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 109/163 2024-01-15T11:23:33.772 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 109/163 2024-01-15T11:23:33.956 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 110/163 2024-01-15T11:23:34.176 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 110/163 2024-01-15T11:23:34.331 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 111/163 2024-01-15T11:23:34.789 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 111/163 2024-01-15T11:23:34.956 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 112/163 2024-01-15T11:23:34.973 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 112/163 2024-01-15T11:23:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 108/163 2024-01-15T11:23:35.228 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /sys 2024-01-15T11:23:35.228 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /proc 2024-01-15T11:23:35.228 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /dev 2024-01-15T11:23:35.228 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /run 2024-01-15T11:23:35.228 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /mnt 2024-01-15T11:23:35.228 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /var/tmp 2024-01-15T11:23:35.228 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /home 2024-01-15T11:23:35.228 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /tmp 2024-01-15T11:23:35.228 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /dev 2024-01-15T11:23:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:23:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 109/163 2024-01-15T11:23:36.741 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 113/163 2024-01-15T11:23:36.925 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 113/163 2024-01-15T11:23:37.062 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.e 114/163 2024-01-15T11:23:37.201 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 109/163 2024-01-15T11:23:37.221 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41. 115/163 2024-01-15T11:23:37.418 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 110/163 2024-01-15T11:23:37.438 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noa 116/163 2024-01-15T11:23:37.640 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 110/163 2024-01-15T11:23:37.661 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noa 116/163 2024-01-15T11:23:37.821 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 111/163 2024-01-15T11:23:37.947 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.n 117/163 2024-01-15T11:23:38.281 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 111/163 2024-01-15T11:23:38.455 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 112/163 2024-01-15T11:23:38.473 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 112/163 2024-01-15T11:23:38.908 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.n 117/163 2024-01-15T11:23:39.104 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 118/163 2024-01-15T11:23:39.197 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 118/163 2024-01-15T11:23:39.539 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el 119/163 2024-01-15T11:23:39.705 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 120/163 2024-01-15T11:23:39.743 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 120/163 2024-01-15T11:23:39.888 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 121/163 2024-01-15T11:23:39.899 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 121/163 2024-01-15T11:23:40.292 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 113/163 2024-01-15T11:23:40.512 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 113/163 2024-01-15T11:23:40.653 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.e 114/163 2024-01-15T11:23:40.871 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41. 115/163 2024-01-15T11:23:41.129 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noa 116/163 2024-01-15T11:23:41.248 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-2:18.0.0-7629.g95c90f41.el8.x86_64 122/163 2024-01-15T11:23:41.349 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noa 116/163 2024-01-15T11:23:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 123/163 2024-01-15T11:23:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 123/163 2024-01-15T11:23:41.541 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.n 117/163 2024-01-15T11:23:41.634 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g95c90 124/163 2024-01-15T11:23:41.677 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g95c90 124/163 2024-01-15T11:23:41.863 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 125/163 2024-01-15T11:23:42.221 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 125/163 2024-01-15T11:23:42.437 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 126/163 2024-01-15T11:23:42.508 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.n 117/163 2024-01-15T11:23:42.703 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 118/163 2024-01-15T11:23:42.798 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 118/163 2024-01-15T11:23:43.176 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el 119/163 2024-01-15T11:23:43.367 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 120/163 2024-01-15T11:23:43.405 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 120/163 2024-01-15T11:23:43.570 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 121/163 2024-01-15T11:23:43.583 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 121/163 2024-01-15T11:23:44.939 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-2:18.0.0-7629.g95c90f41.el8.x86_64 122/163 2024-01-15T11:23:45.141 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 123/163 2024-01-15T11:23:45.177 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 123/163 2024-01-15T11:23:45.385 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g95c90 124/163 2024-01-15T11:23:45.428 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g95c90 124/163 2024-01-15T11:23:45.586 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 125/163 2024-01-15T11:23:45.948 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 125/163 2024-01-15T11:23:46.137 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 126/163 2024-01-15T11:23:47.081 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 126/163 2024-01-15T11:23:48.019 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-test-2:18.0.0-7629.g95c90f41.el8.x86_64 127/163 2024-01-15T11:23:48.316 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/163 2024-01-15T11:23:48.607 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 129/163 2024-01-15T11:23:48.961 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 130/163 2024-01-15T11:23:49.288 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 131/163 2024-01-15T11:23:49.461 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libtool-2.4.6-25.el8.x86_64 132/163 2024-01-15T11:23:49.474 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 132/163 2024-01-15T11:23:49.722 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 133/163 2024-01-15T11:23:49.882 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 134/163 2024-01-15T11:23:50.045 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g95c90f41.el8.x86_ 135/163 2024-01-15T11:23:50.793 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 126/163 2024-01-15T11:23:51.764 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-test-2:18.0.0-7629.g95c90f41.el8.x86_64 127/163 2024-01-15T11:23:52.090 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/163 2024-01-15T11:23:52.298 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-15T11:23:52.423 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 129/163 2024-01-15T11:23:52.594 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-15T11:23:52.632 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-15T11:23:52.822 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 130/163 2024-01-15T11:23:53.032 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 137/163 2024-01-15T11:23:53.170 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 131/163 2024-01-15T11:23:53.402 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libtool-2.4.6-25.el8.x86_64 132/163 2024-01-15T11:23:53.416 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 132/163 2024-01-15T11:23:53.417 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 138/163 2024-01-15T11:23:53.638 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 133/163 2024-01-15T11:23:53.648 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 139/163 2024-01-15T11:23:53.843 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-top-2:18.0.0-7629.g95c90f41.el8.noarch 140/163 2024-01-15T11:23:53.848 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 134/163 2024-01-15T11:23:54.051 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 141/163 2024-01-15T11:23:54.061 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g95c90f41.el8.x86_ 135/163 2024-01-15T11:23:54.303 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-nbd-2:18.0.0-7629.g95c90f41.el8.x86_64 142/163 2024-01-15T11:23:54.595 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 143/163 2024-01-15T11:23:54.793 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flex-2.6.1-9.el8.x86_64 144/163 2024-01-15T11:23:54.868 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 144/163 2024-01-15T11:23:55.110 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : bison-3.0.4-10.el8.x86_64 145/163 2024-01-15T11:23:55.123 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 145/163 2024-01-15T11:23:55.310 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 146/163 2024-01-15T11:23:55.319 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 146/163 2024-01-15T11:23:55.487 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 147/163 2024-01-15T11:23:55.631 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 148/163 2024-01-15T11:23:55.795 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 149/163 2024-01-15T11:23:55.972 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 150/163 2024-01-15T11:23:55.972 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 151/163 2024-01-15T11:23:56.246 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 151/163 2024-01-15T11:23:56.247 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 152/163 2024-01-15T11:23:56.338 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-15T11:23:56.496 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 152/163 2024-01-15T11:23:56.497 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 153/163 2024-01-15T11:23:56.668 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-15T11:23:56.708 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-15T11:23:56.732 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 153/163 2024-01-15T11:23:56.978 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 154/163 2024-01-15T11:23:57.126 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 137/163 2024-01-15T11:23:57.144 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 155/163 2024-01-15T11:23:57.361 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 156/163 2024-01-15T11:23:57.361 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 157/163 2024-01-15T11:23:57.475 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 138/163 2024-01-15T11:23:57.595 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 157/163 2024-01-15T11:23:57.595 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 158/163 2024-01-15T11:23:57.730 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 139/163 2024-01-15T11:23:57.828 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 158/163 2024-01-15T11:23:57.828 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 159/163 2024-01-15T11:23:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : cephfs-top-2:18.0.0-7629.g95c90f41.el8.noarch 140/163 2024-01-15T11:23:58.053 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 159/163 2024-01-15T11:23:58.053 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 160/163 2024-01-15T11:23:58.143 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : rbd-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 141/163 2024-01-15T11:23:58.270 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 160/163 2024-01-15T11:23:58.270 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 161/163 2024-01-15T11:23:58.468 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : rbd-nbd-2:18.0.0-7629.g95c90f41.el8.x86_64 142/163 2024-01-15T11:23:58.511 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 161/163 2024-01-15T11:23:58.678 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 162/163 2024-01-15T11:23:58.678 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 163/163 2024-01-15T11:23:58.752 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 143/163 2024-01-15T11:23:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : flex-2.6.1-9.el8.x86_64 144/163 2024-01-15T11:23:59.019 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 144/163 2024-01-15T11:23:59.227 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : bison-3.0.4-10.el8.x86_64 145/163 2024-01-15T11:23:59.240 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 145/163 2024-01-15T11:23:59.451 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 146/163 2024-01-15T11:23:59.460 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 146/163 2024-01-15T11:23:59.591 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 163/163 2024-01-15T11:23:59.591 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-2:18.0.0-7629.g95c90f41.el8.x86_64 1/163 2024-01-15T11:23:59.591 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 2/163 2024-01-15T11:23:59.591 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 3/163 2024-01-15T11:23:59.591 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 4/163 2024-01-15T11:23:59.591 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g95c90 5/163 2024-01-15T11:23:59.591 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 6/163 2024-01-15T11:23:59.591 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 7/163 2024-01-15T11:23:59.591 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 8/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 9/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 10/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 11/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-test-2:18.0.0-7629.g95c90f41.el8.x86_64 12/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 13/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g95c90f41.el8.x86_ 14/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 15/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 16/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados-devel-2:18.0.0-7629.g95c90f41.el8.x86_6 17/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86 18/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 19/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el 20/163 2024-01-15T11:23:59.592 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g95c90f41.el8. 21/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g95c90f41.el8.x86_6 22/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rados-2:18.0.0-7629.g95c90f41.el8.x86_64 23/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rbd-2:18.0.0-7629.g95c90f41.el8.x86_64 24/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rgw-2:18.0.0-7629.g95c90f41.el8.x86_64 25/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 26/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 27/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g95c90f41.el8.x86_64 28/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41. 29/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noa 30/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.n 31/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 32/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el 33/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 34/163 2024-01-15T11:23:59.593 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.e 35/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 36/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 37/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-top-2:18.0.0-7629.g95c90f41.el8.noarch 38/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : luarocks-3.5.0-1.el8.noarch 42/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 43/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 44/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 45/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 46/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 47/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 48/163 2024-01-15T11:23:59.594 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 49/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 50/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 51/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 52/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 53/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 54/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 55/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 56/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 57/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-portend-2.6-1.el8.noarch 58/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 59/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 60/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 61/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 62/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 63/163 2024-01-15T11:23:59.595 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 64/163 2024-01-15T11:23:59.596 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 65/163 2024-01-15T11:23:59.596 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 66/163 2024-01-15T11:23:59.596 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 67/163 2024-01-15T11:23:59.596 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 68/163 2024-01-15T11:23:59.596 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 69/163 2024-01-15T11:23:59.596 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 70/163 2024-01-15T11:23:59.596 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 71/163 2024-01-15T11:23:59.596 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 72/163 2024-01-15T11:23:59.596 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lua-devel-5.3.4-12.el8.x86_64 73/163 2024-01-15T11:23:59.596 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 74/163 2024-01-15T11:23:59.596 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 75/163 2024-01-15T11:23:59.596 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 76/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 77/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 79/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 80/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 82/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 83/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 84/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 85/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 86/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 87/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 88/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 89/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 90/163 2024-01-15T11:23:59.597 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 91/163 2024-01-15T11:23:59.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 92/163 2024-01-15T11:23:59.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 93/163 2024-01-15T11:23:59.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 94/163 2024-01-15T11:23:59.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 95/163 2024-01-15T11:23:59.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 96/163 2024-01-15T11:23:59.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 97/163 2024-01-15T11:23:59.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 98/163 2024-01-15T11:23:59.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 99/163 2024-01-15T11:23:59.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 100/163 2024-01-15T11:23:59.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 101/163 2024-01-15T11:23:59.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 102/163 2024-01-15T11:23:59.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 103/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 104/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 106/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : bison-3.0.4-10.el8.x86_64 107/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flex-2.6.1-9.el8.x86_64 108/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 109/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 110/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 111/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 112/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 113/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 114/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 115/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 116/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 117/163 2024-01-15T11:23:59.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 118/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 119/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 120/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 121/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 122/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 123/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : automake-1.16.1-8.el8.noarch 124/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libpq-13.11-1.el8.x86_64 125/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : jq-1.6-7.el8.x86_64 126/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 127/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 128/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 129/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 130/163 2024-01-15T11:23:59.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 131/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 132/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 133/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 134/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 135/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 136/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 137/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 138/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 139/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 140/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 141/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 142/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 143/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 144/163 2024-01-15T11:23:59.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 145/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 146/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 147/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 148/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 149/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 150/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 151/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 152/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 153/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 154/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 155/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 156/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 157/163 2024-01-15T11:23:59.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 158/163 2024-01-15T11:23:59.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 159/163 2024-01-15T11:23:59.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 160/163 2024-01-15T11:23:59.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/163 2024-01-15T11:23:59.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 162/163 2024-01-15T11:23:59.603 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 147/163 2024-01-15T11:23:59.784 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 148/163 2024-01-15T11:24:00.011 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 149/163 2024-01-15T11:24:00.171 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 150/163 2024-01-15T11:24:00.172 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 151/163 2024-01-15T11:24:00.462 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 151/163 2024-01-15T11:24:00.462 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 152/163 2024-01-15T11:24:00.705 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 152/163 2024-01-15T11:24:00.705 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 153/163 2024-01-15T11:24:00.958 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 153/163 2024-01-15T11:24:01.202 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 154/163 2024-01-15T11:24:01.395 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 155/163 2024-01-15T11:24:01.619 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 156/163 2024-01-15T11:24:01.619 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 157/163 2024-01-15T11:24:01.878 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 157/163 2024-01-15T11:24:01.878 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 158/163 2024-01-15T11:24:02.120 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 158/163 2024-01-15T11:24:02.120 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 159/163 2024-01-15T11:24:02.361 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 159/163 2024-01-15T11:24:02.361 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 160/163 2024-01-15T11:24:02.569 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 160/163 2024-01-15T11:24:02.570 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 161/163 2024-01-15T11:24:02.811 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 161/163 2024-01-15T11:24:02.978 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 162/163 2024-01-15T11:24:02.978 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 163/163 2024-01-15T11:24:03.930 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 163/163 2024-01-15T11:24:03.931 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-2:18.0.0-7629.g95c90f41.el8.x86_64 1/163 2024-01-15T11:24:03.931 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 2/163 2024-01-15T11:24:03.931 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 3/163 2024-01-15T11:24:03.931 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 4/163 2024-01-15T11:24:03.931 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g95c90 5/163 2024-01-15T11:24:03.931 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 6/163 2024-01-15T11:24:03.931 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 7/163 2024-01-15T11:24:03.931 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 8/163 2024-01-15T11:24:03.931 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 9/163 2024-01-15T11:24:03.931 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 10/163 2024-01-15T11:24:03.931 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 11/163 2024-01-15T11:24:03.931 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-test-2:18.0.0-7629.g95c90f41.el8.x86_64 12/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 13/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g95c90f41.el8.x86_ 14/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 15/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 16/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librados-devel-2:18.0.0-7629.g95c90f41.el8.x86_6 17/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86 18/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 19/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el 20/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g95c90f41.el8. 21/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g95c90f41.el8.x86_6 22/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-rados-2:18.0.0-7629.g95c90f41.el8.x86_64 23/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-rbd-2:18.0.0-7629.g95c90f41.el8.x86_64 24/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-rgw-2:18.0.0-7629.g95c90f41.el8.x86_64 25/163 2024-01-15T11:24:03.932 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 26/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 27/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g95c90f41.el8.x86_64 28/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41. 29/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noa 30/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.n 31/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 32/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el 33/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 34/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.e 35/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 36/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 37/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : cephfs-top-2:18.0.0-7629.g95c90f41.el8.noarch 38/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/163 2024-01-15T11:24:03.933 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : luarocks-3.5.0-1.el8.noarch 42/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 43/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 44/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 45/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 46/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 47/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 48/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 49/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 50/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 51/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 52/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 53/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 54/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 55/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 56/163 2024-01-15T11:24:03.934 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 57/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-portend-2.6-1.el8.noarch 58/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 59/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 60/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 61/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 62/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 63/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 64/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 65/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 66/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 67/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 68/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 69/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 70/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 71/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 72/163 2024-01-15T11:24:03.935 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 73/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 75/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 76/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 77/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 78/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 79/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 80/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 81/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 82/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 83/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 84/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 85/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 86/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 87/163 2024-01-15T11:24:03.936 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 88/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 89/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 90/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 91/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 92/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 93/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 94/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 95/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 96/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 97/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 98/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 99/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : lua-devel-5.3.4-12.el8.x86_64 100/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 101/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 102/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 103/163 2024-01-15T11:24:03.937 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 104/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 106/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : bison-3.0.4-10.el8.x86_64 107/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : flex-2.6.1-9.el8.x86_64 108/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 109/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 110/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 111/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 112/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 113/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 114/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 115/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 116/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 117/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 118/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 119/163 2024-01-15T11:24:03.938 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 120/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 121/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 122/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 123/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : automake-1.16.1-8.el8.noarch 124/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libpq-13.11-1.el8.x86_64 125/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : jq-1.6-7.el8.x86_64 126/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 127/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 128/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 129/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 130/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 131/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 132/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 133/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 134/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 135/163 2024-01-15T11:24:03.939 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 136/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 137/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 138/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 139/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 140/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 141/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 142/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 143/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 144/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 145/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 146/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 147/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 148/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 149/163 2024-01-15T11:24:03.940 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 150/163 2024-01-15T11:24:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 151/163 2024-01-15T11:24:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 152/163 2024-01-15T11:24:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 153/163 2024-01-15T11:24:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 154/163 2024-01-15T11:24:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 155/163 2024-01-15T11:24:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 156/163 2024-01-15T11:24:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 157/163 2024-01-15T11:24:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 158/163 2024-01-15T11:24:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 159/163 2024-01-15T11:24:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 160/163 2024-01-15T11:24:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/163 2024-01-15T11:24:03.941 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 162/163 2024-01-15T11:24:05.849 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 163/163 2024-01-15T11:24:05.849 INFO:teuthology.orchestra.run.smithi113.stdout:Installed products updated. 2024-01-15T11:24:06.465 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:06.465 INFO:teuthology.orchestra.run.smithi113.stdout:Upgraded: 2024-01-15T11:24:06.465 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-15T11:24:06.465 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-15T11:24:06.466 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: automake-1.16.1-8.el8.noarch 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: bison-3.0.4-10.el8.x86_64 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:06.467 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-top-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:06.468 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: flex-2.6.1-9.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: jq-1.6-7.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-15T11:24:06.469 INFO:teuthology.orchestra.run.smithi113.stdout: libpq-13.11-1.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-15T11:24:06.470 INFO:teuthology.orchestra.run.smithi113.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: lua-devel-5.3.4-12.el8.x86_64 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: luarocks-3.5.0-1.el8.noarch 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-15T11:24:06.471 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-15T11:24:06.472 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-15T11:24:06.472 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-15T11:24:06.472 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-15T11:24:06.472 INFO:teuthology.orchestra.run.smithi113.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-15T11:24:06.472 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-15T11:24:06.472 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.472 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.472 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.472 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-15T11:24:06.472 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-15T11:24:06.472 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-15T11:24:06.472 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-15T11:24:06.473 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend-2.6-1.el8.noarch 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.474 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-15T11:24:06.475 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.476 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.476 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:06.476 INFO:teuthology.orchestra.run.smithi113.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-15T11:24:06.476 INFO:teuthology.orchestra.run.smithi113.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-15T11:24:06.476 INFO:teuthology.orchestra.run.smithi113.stdout: uuid-1.6.2-43.el8.x86_64 2024-01-15T11:24:06.476 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-15T11:24:06.476 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-15T11:24:06.477 INFO:teuthology.orchestra.run.smithi113.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-15T11:24:06.477 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:06.477 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-01-15T11:24:06.729 DEBUG:teuthology.parallel:result is None 2024-01-15T11:24:10.785 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 163/163 2024-01-15T11:24:10.786 INFO:teuthology.orchestra.run.smithi141.stdout:Installed products updated. 2024-01-15T11:24:11.843 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:11.843 INFO:teuthology.orchestra.run.smithi141.stdout:Upgraded: 2024-01-15T11:24:11.843 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-15T11:24:11.843 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-15T11:24:11.843 INFO:teuthology.orchestra.run.smithi141.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-15T11:24:11.843 INFO:teuthology.orchestra.run.smithi141.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-15T11:24:11.844 INFO:teuthology.orchestra.run.smithi141.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-15T11:24:11.844 INFO:teuthology.orchestra.run.smithi141.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-15T11:24:11.844 INFO:teuthology.orchestra.run.smithi141.stdout: librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.844 INFO:teuthology.orchestra.run.smithi141.stdout: librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.844 INFO:teuthology.orchestra.run.smithi141.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-15T11:24:11.844 INFO:teuthology.orchestra.run.smithi141.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-15T11:24:11.844 INFO:teuthology.orchestra.run.smithi141.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-15T11:24:11.844 INFO:teuthology.orchestra.run.smithi141.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-15T11:24:11.844 INFO:teuthology.orchestra.run.smithi141.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-15T11:24:11.844 INFO:teuthology.orchestra.run.smithi141.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-15T11:24:11.844 INFO:teuthology.orchestra.run.smithi141.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-15T11:24:11.845 INFO:teuthology.orchestra.run.smithi141.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-15T11:24:11.845 INFO:teuthology.orchestra.run.smithi141.stdout:Installed: 2024-01-15T11:24:11.845 INFO:teuthology.orchestra.run.smithi141.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-15T11:24:11.845 INFO:teuthology.orchestra.run.smithi141.stdout: automake-1.16.1-8.el8.noarch 2024-01-15T11:24:11.845 INFO:teuthology.orchestra.run.smithi141.stdout: bison-3.0.4-10.el8.x86_64 2024-01-15T11:24:11.845 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.845 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.845 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.845 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.845 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:11.845 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.846 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.846 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.846 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:11.846 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:11.846 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:11.846 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:11.846 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:11.846 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.846 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.846 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:11.846 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.846 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.847 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-test-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.847 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:11.847 INFO:teuthology.orchestra.run.smithi141.stdout: cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:11.847 INFO:teuthology.orchestra.run.smithi141.stdout: cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.848 INFO:teuthology.orchestra.run.smithi141.stdout: cephfs-top-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T11:24:11.848 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-15T11:24:11.848 INFO:teuthology.orchestra.run.smithi141.stdout: flex-2.6.1-9.el8.x86_64 2024-01-15T11:24:11.848 INFO:teuthology.orchestra.run.smithi141.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-15T11:24:11.848 INFO:teuthology.orchestra.run.smithi141.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-15T11:24:11.848 INFO:teuthology.orchestra.run.smithi141.stdout: jq-1.6-7.el8.x86_64 2024-01-15T11:24:11.848 INFO:teuthology.orchestra.run.smithi141.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-15T11:24:11.848 INFO:teuthology.orchestra.run.smithi141.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-15T11:24:11.848 INFO:teuthology.orchestra.run.smithi141.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-15T11:24:11.848 INFO:teuthology.orchestra.run.smithi141.stdout: libcephfs-devel-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.848 INFO:teuthology.orchestra.run.smithi141.stdout: libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.849 INFO:teuthology.orchestra.run.smithi141.stdout: libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.849 INFO:teuthology.orchestra.run.smithi141.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-15T11:24:11.849 INFO:teuthology.orchestra.run.smithi141.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-15T11:24:11.849 INFO:teuthology.orchestra.run.smithi141.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-15T11:24:11.849 INFO:teuthology.orchestra.run.smithi141.stdout: libpq-13.11-1.el8.x86_64 2024-01-15T11:24:11.849 INFO:teuthology.orchestra.run.smithi141.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-15T11:24:11.849 INFO:teuthology.orchestra.run.smithi141.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-15T11:24:11.849 INFO:teuthology.orchestra.run.smithi141.stdout: librados-devel-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.849 INFO:teuthology.orchestra.run.smithi141.stdout: libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.849 INFO:teuthology.orchestra.run.smithi141.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-15T11:24:11.849 INFO:teuthology.orchestra.run.smithi141.stdout: librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.849 INFO:teuthology.orchestra.run.smithi141.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-15T11:24:11.850 INFO:teuthology.orchestra.run.smithi141.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-15T11:24:11.850 INFO:teuthology.orchestra.run.smithi141.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-15T11:24:11.850 INFO:teuthology.orchestra.run.smithi141.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-15T11:24:11.850 INFO:teuthology.orchestra.run.smithi141.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-15T11:24:11.850 INFO:teuthology.orchestra.run.smithi141.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-15T11:24:11.850 INFO:teuthology.orchestra.run.smithi141.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-15T11:24:11.850 INFO:teuthology.orchestra.run.smithi141.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-15T11:24:11.850 INFO:teuthology.orchestra.run.smithi141.stdout: lua-devel-5.3.4-12.el8.x86_64 2024-01-15T11:24:11.850 INFO:teuthology.orchestra.run.smithi141.stdout: luarocks-3.5.0-1.el8.noarch 2024-01-15T11:24:11.850 INFO:teuthology.orchestra.run.smithi141.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-15T11:24:11.851 INFO:teuthology.orchestra.run.smithi141.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-15T11:24:11.851 INFO:teuthology.orchestra.run.smithi141.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-15T11:24:11.851 INFO:teuthology.orchestra.run.smithi141.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-15T11:24:11.851 INFO:teuthology.orchestra.run.smithi141.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-15T11:24:11.851 INFO:teuthology.orchestra.run.smithi141.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-15T11:24:11.851 INFO:teuthology.orchestra.run.smithi141.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-15T11:24:11.851 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-15T11:24:11.851 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-15T11:24:11.852 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-15T11:24:11.852 INFO:teuthology.orchestra.run.smithi141.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-15T11:24:11.852 INFO:teuthology.orchestra.run.smithi141.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-15T11:24:11.852 INFO:teuthology.orchestra.run.smithi141.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-15T11:24:11.852 INFO:teuthology.orchestra.run.smithi141.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-15T11:24:11.852 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-15T11:24:11.852 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.852 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.852 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cephfs-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.852 INFO:teuthology.orchestra.run.smithi141.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-15T11:24:11.852 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-15T11:24:11.853 INFO:teuthology.orchestra.run.smithi141.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-portend-2.6-1.el8.noarch 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-15T11:24:11.854 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rados-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rbd-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rgw-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-15T11:24:11.855 INFO:teuthology.orchestra.run.smithi141.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-15T11:24:11.856 INFO:teuthology.orchestra.run.smithi141.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-15T11:24:11.856 INFO:teuthology.orchestra.run.smithi141.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-15T11:24:11.856 INFO:teuthology.orchestra.run.smithi141.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-15T11:24:11.856 INFO:teuthology.orchestra.run.smithi141.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-15T11:24:11.856 INFO:teuthology.orchestra.run.smithi141.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-15T11:24:11.856 INFO:teuthology.orchestra.run.smithi141.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-15T11:24:11.856 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.856 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.856 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-nbd-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T11:24:11.856 INFO:teuthology.orchestra.run.smithi141.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-15T11:24:11.856 INFO:teuthology.orchestra.run.smithi141.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-15T11:24:11.856 INFO:teuthology.orchestra.run.smithi141.stdout: uuid-1.6.2-43.el8.x86_64 2024-01-15T11:24:11.857 INFO:teuthology.orchestra.run.smithi141.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-15T11:24:11.857 INFO:teuthology.orchestra.run.smithi141.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-15T11:24:11.857 INFO:teuthology.orchestra.run.smithi141.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-15T11:24:11.857 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:11.857 INFO:teuthology.orchestra.run.smithi141.stdout:Complete! 2024-01-15T11:24:12.062 DEBUG:teuthology.parallel:result is None 2024-01-15T11:24:12.063 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T11:24:12.321 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-15T11:24:12.351 INFO:teuthology.orchestra.run.smithi113.stdout:18.0.0-7629.g95c90f41.el8 2024-01-15T11:24:12.351 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g95c90f41.el8 2024-01-15T11:24:12.352 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g95c90f41 is installed. 2024-01-15T11:24:12.353 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T11:24:12.588 DEBUG:teuthology.orchestra.run.smithi141:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-15T11:24:12.615 INFO:teuthology.orchestra.run.smithi141.stdout:18.0.0-7629.g95c90f41.el8 2024-01-15T11:24:12.615 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g95c90f41.el8 2024-01-15T11:24:12.616 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g95c90f41 is installed. 2024-01-15T11:24:12.617 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-15T11:24:12.618 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:12.618 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-15T11:24:12.666 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:12.666 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-15T11:24:12.710 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-15T11:24:12.711 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:12.711 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/daemon-helper 2024-01-15T11:24:12.742 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-15T11:24:12.814 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:12.814 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/usr/bin/daemon-helper 2024-01-15T11:24:12.843 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-15T11:24:12.912 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-15T11:24:12.912 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:12.912 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-15T11:24:12.944 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-15T11:24:13.015 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:13.015 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-15T11:24:13.044 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-15T11:24:13.112 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-15T11:24:13.112 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:13.112 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/stdin-killer 2024-01-15T11:24:13.143 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-15T11:24:13.212 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:13.212 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/usr/bin/stdin-killer 2024-01-15T11:24:13.240 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-15T11:24:13.306 INFO:teuthology.run_tasks:Running task ceph... 2024-01-15T11:24:13.396 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-15T11:24:13.397 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 777 /var/log/ceph 2024-01-15T11:24:13.399 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 777 /var/log/ceph 2024-01-15T11:24:13.428 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-15T11:24:13.428 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-15T11:24:13.472 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-15T11:24:13.500 INFO:tasks.ceph:Creating extra log directories... 2024-01-15T11:24:13.500 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-15T11:24:13.543 DEBUG:teuthology.orchestra.run.smithi141:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-15T11:24:13.596 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-01-15T11:24:13.596 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-01-15T11:24:13.596 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-01-15T11:24:13.596 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:13.596 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-15T11:24:13.615 DEBUG:teuthology.orchestra.run.smithi113:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-15T11:24:13.732 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-15T11:24:13.760 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:13.760 DEBUG:teuthology.orchestra.run.smithi141:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-15T11:24:13.778 DEBUG:teuthology.orchestra.run.smithi141:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-15T11:24:13.891 DEBUG:teuthology.orchestra.run.smithi141:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-15T11:24:13.924 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-15T11:24:13.924 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-15T11:24:13.924 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517064', 'branch': 'rishabh-clone-stats', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}}', 'email': None, 'first_in_suite': False, 'job_id': '7517064', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': 'c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'rishabh-clone-stats'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'mgr-vol-clone-stats-temp', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 9877, 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'mgr-vol-clone-stats-temp', 'suite_path': '/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/rishabh-d-dave/ceph', 'suite_sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi113.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLCA/CGEJo18u9e0CDj25sBHw8iC1BuaeVgBQ5ju1ZHJYeitElGPXNnh/2Fwdt41tcUkHPs+17DXH3op1xm/0xM=', 'smithi141.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8dkBsd2zlYWrvj7QYzhElnvOE/iuNKCyODK6mvmlcRLi0HkL8ocHk57GidyxP8BMY8jl31lLozx7XyO3GSprE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': 'c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestCloneProgress']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-15_11:07:32', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-15T11:24:13.925 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-15T11:24:13.927 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-15T11:24:13.975 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m0777 -- /var/run/ceph 2024-01-15T11:24:13.987 DEBUG:teuthology.orchestra.run.smithi141:> sudo install -d -m0777 -- /var/run/ceph 2024-01-15T11:24:14.042 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:14.042 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T11:24:14.074 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-15T11:24:14.074 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_1 2024-01-15T11:24:14.133 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-15T11:24:14.133 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T11:24:14.133 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 6h/6d Inode: 175022 Links: 1 2024-01-15T11:24:14.133 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T11:24:14.133 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T11:24:14.133 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-01-15 11:23:59.233703493 +0000 2024-01-15T11:24:14.133 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-01-15 11:21:05.875899358 +0000 2024-01-15T11:24:14.134 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-01-15 11:21:05.875899358 +0000 2024-01-15T11:24:14.134 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-01-15T11:24:14.134 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-15T11:24:14.201 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-01-15T11:24:14.201 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-01-15T11:24:14.201 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000235008 s, 2.2 MB/s 2024-01-15T11:24:14.202 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-15T11:24:14.262 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_2 2024-01-15T11:24:14.321 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-15T11:24:14.322 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T11:24:14.322 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 6h/6d Inode: 176399 Links: 1 2024-01-15T11:24:14.322 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T11:24:14.322 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T11:24:14.322 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-01-15 11:23:59.233703493 +0000 2024-01-15T11:24:14.322 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-01-15 11:21:06.241891317 +0000 2024-01-15T11:24:14.322 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-01-15 11:21:06.241891317 +0000 2024-01-15T11:24:14.322 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-01-15T11:24:14.322 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-15T11:24:14.389 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-01-15T11:24:14.389 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-01-15T11:24:14.389 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000199392 s, 2.6 MB/s 2024-01-15T11:24:14.390 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-15T11:24:14.449 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_3 2024-01-15T11:24:14.508 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-15T11:24:14.508 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T11:24:14.508 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 6h/6d Inode: 176501 Links: 1 2024-01-15T11:24:14.508 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T11:24:14.508 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T11:24:14.508 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-01-15 11:23:59.233703493 +0000 2024-01-15T11:24:14.508 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-01-15 11:21:06.602883386 +0000 2024-01-15T11:24:14.508 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-01-15 11:21:06.602883386 +0000 2024-01-15T11:24:14.508 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-01-15T11:24:14.509 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-15T11:24:14.575 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-01-15T11:24:14.575 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-01-15T11:24:14.575 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000202995 s, 2.5 MB/s 2024-01-15T11:24:14.576 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-15T11:24:14.634 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_4 2024-01-15T11:24:14.693 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-15T11:24:14.693 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T11:24:14.693 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 6h/6d Inode: 172936 Links: 1 2024-01-15T11:24:14.693 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T11:24:14.693 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T11:24:14.693 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-01-15 11:23:59.233703493 +0000 2024-01-15T11:24:14.693 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-01-15 11:21:06.934876091 +0000 2024-01-15T11:24:14.693 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-01-15 11:21:06.934876091 +0000 2024-01-15T11:24:14.693 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-01-15T11:24:14.693 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-15T11:24:14.759 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-01-15T11:24:14.759 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-01-15T11:24:14.759 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000305516 s, 1.7 MB/s 2024-01-15T11:24:14.760 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-15T11:24:14.817 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T11:24:14.817 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:14.818 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T11:24:14.839 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-15T11:24:14.839 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_1 2024-01-15T11:24:14.900 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-15T11:24:14.900 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T11:24:14.900 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 175558 Links: 1 2024-01-15T11:24:14.900 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T11:24:14.900 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T11:24:14.900 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-15 11:24:03.543062043 +0000 2024-01-15T11:24:14.900 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-15 11:21:05.875083730 +0000 2024-01-15T11:24:14.900 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-15 11:21:05.875083730 +0000 2024-01-15T11:24:14.901 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2024-01-15T11:24:14.901 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-15T11:24:14.970 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-15T11:24:14.970 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-15T11:24:14.971 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000243499 s, 2.1 MB/s 2024-01-15T11:24:14.972 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-15T11:24:15.032 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_2 2024-01-15T11:24:15.091 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-15T11:24:15.092 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T11:24:15.092 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 173803 Links: 1 2024-01-15T11:24:15.092 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T11:24:15.092 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T11:24:15.092 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-15 11:24:03.543062043 +0000 2024-01-15T11:24:15.092 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-15 11:21:06.216076373 +0000 2024-01-15T11:24:15.092 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-15 11:21:06.216076373 +0000 2024-01-15T11:24:15.092 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2024-01-15T11:24:15.093 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-15T11:24:15.159 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-15T11:24:15.159 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-15T11:24:15.159 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000223136 s, 2.3 MB/s 2024-01-15T11:24:15.160 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-15T11:24:15.220 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_3 2024-01-15T11:24:15.278 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-15T11:24:15.278 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T11:24:15.278 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 173878 Links: 1 2024-01-15T11:24:15.278 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T11:24:15.278 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T11:24:15.279 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-15 11:24:03.544062022 +0000 2024-01-15T11:24:15.279 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-15 11:21:06.552069123 +0000 2024-01-15T11:24:15.279 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-15 11:21:06.552069123 +0000 2024-01-15T11:24:15.279 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2024-01-15T11:24:15.279 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-15T11:24:15.345 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-15T11:24:15.345 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-15T11:24:15.345 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000194803 s, 2.6 MB/s 2024-01-15T11:24:15.347 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-15T11:24:15.405 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_4 2024-01-15T11:24:15.464 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-15T11:24:15.464 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T11:24:15.464 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 173935 Links: 1 2024-01-15T11:24:15.464 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T11:24:15.464 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T11:24:15.464 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-15 11:24:03.544062022 +0000 2024-01-15T11:24:15.464 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-15 11:21:06.917061248 +0000 2024-01-15T11:24:15.464 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-15 11:21:06.917061248 +0000 2024-01-15T11:24:15.464 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2024-01-15T11:24:15.465 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-15T11:24:15.529 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-15T11:24:15.530 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-15T11:24:15.530 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000209821 s, 2.4 MB/s 2024-01-15T11:24:15.531 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-15T11:24:15.588 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T11:24:15.589 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi113.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi141.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-15T11:24:15.589 INFO:tasks.ceph:Generating config... 2024-01-15T11:24:15.590 INFO:tasks.ceph:[client] client mount timeout = 600 2024-01-15T11:24:15.590 INFO:tasks.ceph:[client] debug client = 20 2024-01-15T11:24:15.590 INFO:tasks.ceph:[client] debug ms = 1 2024-01-15T11:24:15.590 INFO:tasks.ceph:[client] pid file = 2024-01-15T11:24:15.591 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-01-15T11:24:15.591 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-01-15T11:24:15.591 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-01-15T11:24:15.591 INFO:tasks.ceph:[mds] debug mds = 20 2024-01-15T11:24:15.591 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-01-15T11:24:15.591 INFO:tasks.ceph:[mds] debug ms = 1 2024-01-15T11:24:15.591 INFO:tasks.ceph:[mds] mds debug frag = True 2024-01-15T11:24:15.591 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-01-15T11:24:15.591 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-01-15T11:24:15.591 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-01-15T11:24:15.591 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-01-15T11:24:15.592 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-01-15T11:24:15.592 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-01-15T11:24:15.592 INFO:tasks.ceph:[mgr] debug client = 20 2024-01-15T11:24:15.592 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-01-15T11:24:15.592 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-15T11:24:15.592 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-15T11:24:15.592 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-15T11:24:15.592 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-15T11:24:15.592 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-15T11:24:15.593 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-01-15T11:24:15.593 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-15T11:24:15.593 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-15T11:24:15.593 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-15T11:24:15.593 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-15T11:24:15.593 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-15T11:24:15.593 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-01-15T11:24:15.593 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-01-15T11:24:15.593 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-15T11:24:15.593 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-15T11:24:15.593 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-01-15T11:24:15.593 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-15T11:24:15.594 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-15T11:24:15.594 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-15T11:24:15.594 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-15T11:24:15.594 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-15T11:24:15.594 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-01-15T11:24:15.594 INFO:tasks.ceph:Setting up mon.a... 2024-01-15T11:24:15.594 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-15T11:24:15.659 INFO:teuthology.orchestra.run.smithi113.stdout:creating /etc/ceph/ceph.keyring 2024-01-15T11:24:15.663 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-15T11:24:15.712 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-15T11:24:15.740 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.113'), ('mon.b', '172.21.15.141'), ('mon.c', '[v2:172.21.15.141:3301,v1:172.21.15.141:6790]')] 2024-01-15T11:24:15.740 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.113,172.21.15.141,[v2:172.21.15.141:3301,v1:172.21.15.141:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-01-15T11:24:15.741 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:15.741 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-15T11:24:15.798 DEBUG:teuthology.orchestra.run.smithi113:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.113 --add b 172.21.15.141 --addv c '[v2:172.21.15.141:3301,v1:172.21.15.141:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:15.890 INFO:teuthology.orchestra.run.smithi113.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:15.890 INFO:teuthology.orchestra.run.smithi113.stdout:monmaptool: generated fsid 910f8628-cd75-4a37-bf4e-e3603a95290d 2024-01-15T11:24:15.890 INFO:teuthology.orchestra.run.smithi113.stdout:setting min_mon_release = pacific 2024-01-15T11:24:15.890 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 0 2024-01-15T11:24:15.890 INFO:teuthology.orchestra.run.smithi113.stdout:fsid 910f8628-cd75-4a37-bf4e-e3603a95290d 2024-01-15T11:24:15.890 INFO:teuthology.orchestra.run.smithi113.stdout:last_changed 2024-01-15T11:24:15.893382+0000 2024-01-15T11:24:15.891 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-01-15T11:24:15.893382+0000 2024-01-15T11:24:15.891 INFO:teuthology.orchestra.run.smithi113.stdout:min_mon_release 16 (pacific) 2024-01-15T11:24:15.891 INFO:teuthology.orchestra.run.smithi113.stdout:election_strategy: 1 2024-01-15T11:24:15.891 INFO:teuthology.orchestra.run.smithi113.stdout:0: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.a 2024-01-15T11:24:15.891 INFO:teuthology.orchestra.run.smithi113.stdout:1: [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] mon.b 2024-01-15T11:24:15.891 INFO:teuthology.orchestra.run.smithi113.stdout:2: [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] mon.c 2024-01-15T11:24:15.891 INFO:teuthology.orchestra.run.smithi113.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-15T11:24:15.892 DEBUG:teuthology.orchestra.run.smithi113:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-15T11:24:15.909 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 910f8628-cd75-4a37-bf4e-e3603a95290d... 2024-01-15T11:24:15.910 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-15T11:24:15.952 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-15T11:24:15.995 INFO:teuthology.orchestra.run.smithi113.stdout:[global] 2024-01-15T11:24:15.995 INFO:teuthology.orchestra.run.smithi113.stdout: chdir = "" 2024-01-15T11:24:15.995 INFO:teuthology.orchestra.run.smithi113.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-15T11:24:15.995 INFO:teuthology.orchestra.run.smithi113.stdout: auth supported = cephx 2024-01-15T11:24:15.996 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.996 INFO:teuthology.orchestra.run.smithi113.stdout: filestore xattr use omap = true 2024-01-15T11:24:15.996 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.996 INFO:teuthology.orchestra.run.smithi113.stdout: mon clock drift allowed = 1.000 2024-01-15T11:24:15.996 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.996 INFO:teuthology.orchestra.run.smithi113.stdout: osd crush chooseleaf type = 0 2024-01-15T11:24:15.996 INFO:teuthology.orchestra.run.smithi113.stdout: auth debug = true 2024-01-15T11:24:15.996 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.996 INFO:teuthology.orchestra.run.smithi113.stdout: ms die on old message = true 2024-01-15T11:24:15.996 INFO:teuthology.orchestra.run.smithi113.stdout: ms die on bug = true 2024-01-15T11:24:15.996 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: mon pg warn max object skew = 0 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: # disable pg_autoscaler by default for new pools 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: osd pool default size = 2 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd allow primary affinity = true 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd allow pg remap = true 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on legacy crush tunables = false 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on crush straw calc version zero = false 2024-01-15T11:24:15.997 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on no sortbitwise = false 2024-01-15T11:24:15.998 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on osd down out interval zero = false 2024-01-15T11:24:15.998 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on too few osds = false 2024-01-15T11:24:15.998 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-15T11:24:15.998 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-15T11:24:15.998 INFO:teuthology.orchestra.run.smithi113.stdout: mon_allow_pool_size_one = true 2024-01-15T11:24:15.998 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.998 INFO:teuthology.orchestra.run.smithi113.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-15T11:24:15.998 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.998 INFO:teuthology.orchestra.run.smithi113.stdout: osd default data pool replay window = 5 2024-01-15T11:24:15.998 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.998 INFO:teuthology.orchestra.run.smithi113.stdout: mon allow pool delete = true 2024-01-15T11:24:15.998 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.999 INFO:teuthology.orchestra.run.smithi113.stdout: mon cluster log file level = debug 2024-01-15T11:24:15.999 INFO:teuthology.orchestra.run.smithi113.stdout: debug asserts on shutdown = true 2024-01-15T11:24:15.999 INFO:teuthology.orchestra.run.smithi113.stdout: mon health detail to clog = false 2024-01-15T11:24:15.999 INFO:teuthology.orchestra.run.smithi113.stdout: mon host = "172.21.15.113,172.21.15.141,[v2:172.21.15.141:3301,v1:172.21.15.141:6790]" 2024-01-15T11:24:15.999 INFO:teuthology.orchestra.run.smithi113.stdout: ms die on skipped message = False 2024-01-15T11:24:15.999 INFO:teuthology.orchestra.run.smithi113.stdout: fsid = 910f8628-cd75-4a37-bf4e-e3603a95290d 2024-01-15T11:24:15.999 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.999 INFO:teuthology.orchestra.run.smithi113.stdout:[osd] 2024-01-15T11:24:15.999 INFO:teuthology.orchestra.run.smithi113.stdout: osd journal size = 100 2024-01-15T11:24:15.999 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:15.999 INFO:teuthology.orchestra.run.smithi113.stdout: osd scrub load threshold = 5.0 2024-01-15T11:24:15.999 INFO:teuthology.orchestra.run.smithi113.stdout: osd scrub max interval = 600 2024-01-15T11:24:16.000 INFO:teuthology.orchestra.run.smithi113.stdout: osd mclock profile = high_recovery_ops 2024-01-15T11:24:16.000 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.000 INFO:teuthology.orchestra.run.smithi113.stdout: osd recover clone overlap = true 2024-01-15T11:24:16.000 INFO:teuthology.orchestra.run.smithi113.stdout: osd recovery max chunk = 1048576 2024-01-15T11:24:16.000 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.000 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug shutdown = true 2024-01-15T11:24:16.000 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug op order = true 2024-01-15T11:24:16.000 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug verify stray on activate = true 2024-01-15T11:24:16.000 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug trim objects = true 2024-01-15T11:24:16.000 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.000 INFO:teuthology.orchestra.run.smithi113.stdout: osd open classes on start = true 2024-01-15T11:24:16.001 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug pg log writeout = true 2024-01-15T11:24:16.001 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.001 INFO:teuthology.orchestra.run.smithi113.stdout: osd deep scrub update digest min age = 30 2024-01-15T11:24:16.001 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.001 INFO:teuthology.orchestra.run.smithi113.stdout: osd map max advance = 10 2024-01-15T11:24:16.001 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.001 INFO:teuthology.orchestra.run.smithi113.stdout: journal zero on create = true 2024-01-15T11:24:16.003 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.003 INFO:teuthology.orchestra.run.smithi113.stdout: filestore ondisk finisher threads = 3 2024-01-15T11:24:16.003 INFO:teuthology.orchestra.run.smithi113.stdout: filestore apply finisher threads = 3 2024-01-15T11:24:16.004 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.004 INFO:teuthology.orchestra.run.smithi113.stdout: bdev debug aio = true 2024-01-15T11:24:16.004 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug misdirected ops = true 2024-01-15T11:24:16.004 INFO:teuthology.orchestra.run.smithi113.stdout: bdev async discard = True 2024-01-15T11:24:16.004 INFO:teuthology.orchestra.run.smithi113.stdout: bdev enable discard = True 2024-01-15T11:24:16.004 INFO:teuthology.orchestra.run.smithi113.stdout: bluestore allocator = bitmap 2024-01-15T11:24:16.004 INFO:teuthology.orchestra.run.smithi113.stdout: bluestore block size = 96636764160 2024-01-15T11:24:16.004 INFO:teuthology.orchestra.run.smithi113.stdout: bluestore fsck on mount = True 2024-01-15T11:24:16.004 INFO:teuthology.orchestra.run.smithi113.stdout: debug bluefs = 1/20 2024-01-15T11:24:16.004 INFO:teuthology.orchestra.run.smithi113.stdout: debug bluestore = 1/20 2024-01-15T11:24:16.004 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-01-15T11:24:16.004 INFO:teuthology.orchestra.run.smithi113.stdout: debug osd = 20 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout: debug rocksdb = 4/10 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd backfillfull_ratio = 0.85 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd full ratio = 0.9 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd nearfull ratio = 0.8 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout: osd failsafe full ratio = 0.95 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout: osd objectstore = bluestore 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout: osd op complaint time = 180 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout:[mgr] 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout: debug mgr = 20 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout: debug mon = 20 2024-01-15T11:24:16.005 INFO:teuthology.orchestra.run.smithi113.stdout: debug auth = 20 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min pgs per osd = 4 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min bytes per osd = 10 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout: mgr/telemetry/nag = false 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout: debug client = 20 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout: debug finisher = 20 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout:[mon] 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout: debug mon = 20 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout: debug paxos = 20 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout: debug auth = 20 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout: mon data avail warn = 5 2024-01-15T11:24:16.006 INFO:teuthology.orchestra.run.smithi113.stdout: mon mgr mkfs grace = 240 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min pgs per osd = 4 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd reporter subtree level = osd 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd prime pg temp = true 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min bytes per osd = 10 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: auth mon ticket ttl = 660 # 11m 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: auth service ticket ttl = 240 # 4m 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: # don't complain about insecure global_id in the test suite 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-15T11:24:16.007 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.008 INFO:teuthology.orchestra.run.smithi113.stdout: # 1m isn't quite enough 2024-01-15T11:24:16.008 INFO:teuthology.orchestra.run.smithi113.stdout: mon_down_mkfs_grace = 2m 2024-01-15T11:24:16.008 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.008 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_filestore_osds = false 2024-01-15T11:24:16.008 INFO:teuthology.orchestra.run.smithi113.stdout: mon op complaint time = 120 2024-01-15T11:24:16.008 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:16.008 INFO:teuthology.orchestra.run.smithi113.stdout:[client] 2024-01-15T11:24:16.008 INFO:teuthology.orchestra.run.smithi113.stdout: rgw cache enabled = true 2024-01-15T11:24:16.008 INFO:teuthology.orchestra.run.smithi113.stdout: rgw enable ops log = true 2024-01-15T11:24:16.008 INFO:teuthology.orchestra.run.smithi113.stdout: rgw enable usage log = true 2024-01-15T11:24:16.008 INFO:teuthology.orchestra.run.smithi113.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout: client mount timeout = 600 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout: debug client = 20 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout: pid file = "" 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout: rados mon op timeout = 900 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout: rados osd op timeout = 900 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout:[mon.a] 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout:[mon.b] 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout:[mon.c] 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.a] 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.c] 2024-01-15T11:24:16.009 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.b] 2024-01-15T11:24:16.010 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.d] 2024-01-15T11:24:16.010 INFO:teuthology.orchestra.run.smithi113.stdout:[mds] 2024-01-15T11:24:16.010 INFO:teuthology.orchestra.run.smithi113.stdout: debug mds = 20 2024-01-15T11:24:16.010 INFO:teuthology.orchestra.run.smithi113.stdout: debug mds balancer = 20 2024-01-15T11:24:16.010 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-01-15T11:24:16.010 INFO:teuthology.orchestra.run.smithi113.stdout: mds debug frag = True 2024-01-15T11:24:16.010 INFO:teuthology.orchestra.run.smithi113.stdout: mds debug scatterstat = True 2024-01-15T11:24:16.010 INFO:teuthology.orchestra.run.smithi113.stdout: mds op complaint time = 180 2024-01-15T11:24:16.010 INFO:teuthology.orchestra.run.smithi113.stdout: mds verify scatter = True 2024-01-15T11:24:16.010 INFO:teuthology.orchestra.run.smithi113.stdout: osd op complaint time = 180 2024-01-15T11:24:16.010 INFO:teuthology.orchestra.run.smithi113.stdout: rados mon op timeout = 900 2024-01-15T11:24:16.010 INFO:teuthology.orchestra.run.smithi113.stdout: rados osd op timeout = 900 2024-01-15T11:24:16.013 INFO:teuthology.orchestra.run.smithi141.stdout:[global] 2024-01-15T11:24:16.013 INFO:teuthology.orchestra.run.smithi141.stdout: chdir = "" 2024-01-15T11:24:16.013 INFO:teuthology.orchestra.run.smithi141.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-15T11:24:16.013 INFO:teuthology.orchestra.run.smithi141.stdout: auth supported = cephx 2024-01-15T11:24:16.014 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.014 INFO:teuthology.orchestra.run.smithi141.stdout: filestore xattr use omap = true 2024-01-15T11:24:16.014 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.014 INFO:teuthology.orchestra.run.smithi141.stdout: mon clock drift allowed = 1.000 2024-01-15T11:24:16.014 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.015 INFO:teuthology.orchestra.run.smithi141.stdout: osd crush chooseleaf type = 0 2024-01-15T11:24:16.015 INFO:teuthology.orchestra.run.smithi141.stdout: auth debug = true 2024-01-15T11:24:16.015 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.015 INFO:teuthology.orchestra.run.smithi141.stdout: ms die on old message = true 2024-01-15T11:24:16.015 INFO:teuthology.orchestra.run.smithi141.stdout: ms die on bug = true 2024-01-15T11:24:16.016 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.016 INFO:teuthology.orchestra.run.smithi141.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-15T11:24:16.016 INFO:teuthology.orchestra.run.smithi141.stdout: mon pg warn max object skew = 0 2024-01-15T11:24:16.016 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.016 INFO:teuthology.orchestra.run.smithi141.stdout: # disable pg_autoscaler by default for new pools 2024-01-15T11:24:16.017 INFO:teuthology.orchestra.run.smithi141.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-15T11:24:16.017 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.017 INFO:teuthology.orchestra.run.smithi141.stdout: osd pool default size = 2 2024-01-15T11:24:16.017 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.017 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd allow primary affinity = true 2024-01-15T11:24:16.018 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd allow pg remap = true 2024-01-15T11:24:16.018 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on legacy crush tunables = false 2024-01-15T11:24:16.018 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on crush straw calc version zero = false 2024-01-15T11:24:16.018 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on no sortbitwise = false 2024-01-15T11:24:16.018 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on osd down out interval zero = false 2024-01-15T11:24:16.018 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on too few osds = false 2024-01-15T11:24:16.019 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-15T11:24:16.019 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-15T11:24:16.019 INFO:teuthology.orchestra.run.smithi141.stdout: mon_allow_pool_size_one = true 2024-01-15T11:24:16.019 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.019 INFO:teuthology.orchestra.run.smithi141.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-15T11:24:16.020 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.020 INFO:teuthology.orchestra.run.smithi141.stdout: osd default data pool replay window = 5 2024-01-15T11:24:16.020 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.020 INFO:teuthology.orchestra.run.smithi141.stdout: mon allow pool delete = true 2024-01-15T11:24:16.020 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.021 INFO:teuthology.orchestra.run.smithi141.stdout: mon cluster log file level = debug 2024-01-15T11:24:16.021 INFO:teuthology.orchestra.run.smithi141.stdout: debug asserts on shutdown = true 2024-01-15T11:24:16.021 INFO:teuthology.orchestra.run.smithi141.stdout: mon health detail to clog = false 2024-01-15T11:24:16.021 INFO:teuthology.orchestra.run.smithi141.stdout: mon host = "172.21.15.113,172.21.15.141,[v2:172.21.15.141:3301,v1:172.21.15.141:6790]" 2024-01-15T11:24:16.021 INFO:teuthology.orchestra.run.smithi141.stdout: ms die on skipped message = False 2024-01-15T11:24:16.022 INFO:teuthology.orchestra.run.smithi141.stdout: fsid = 910f8628-cd75-4a37-bf4e-e3603a95290d 2024-01-15T11:24:16.022 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.022 INFO:teuthology.orchestra.run.smithi141.stdout:[osd] 2024-01-15T11:24:16.022 INFO:teuthology.orchestra.run.smithi141.stdout: osd journal size = 100 2024-01-15T11:24:16.022 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.022 INFO:teuthology.orchestra.run.smithi141.stdout: osd scrub load threshold = 5.0 2024-01-15T11:24:16.022 INFO:teuthology.orchestra.run.smithi141.stdout: osd scrub max interval = 600 2024-01-15T11:24:16.022 INFO:teuthology.orchestra.run.smithi141.stdout: osd mclock profile = high_recovery_ops 2024-01-15T11:24:16.023 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.023 INFO:teuthology.orchestra.run.smithi141.stdout: osd recover clone overlap = true 2024-01-15T11:24:16.023 INFO:teuthology.orchestra.run.smithi141.stdout: osd recovery max chunk = 1048576 2024-01-15T11:24:16.023 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.024 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug shutdown = true 2024-01-15T11:24:16.024 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug op order = true 2024-01-15T11:24:16.024 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug verify stray on activate = true 2024-01-15T11:24:16.024 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug trim objects = true 2024-01-15T11:24:16.024 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.024 INFO:teuthology.orchestra.run.smithi141.stdout: osd open classes on start = true 2024-01-15T11:24:16.025 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug pg log writeout = true 2024-01-15T11:24:16.025 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.025 INFO:teuthology.orchestra.run.smithi141.stdout: osd deep scrub update digest min age = 30 2024-01-15T11:24:16.025 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.025 INFO:teuthology.orchestra.run.smithi141.stdout: osd map max advance = 10 2024-01-15T11:24:16.026 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.026 INFO:teuthology.orchestra.run.smithi141.stdout: journal zero on create = true 2024-01-15T11:24:16.026 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.026 INFO:teuthology.orchestra.run.smithi141.stdout: filestore ondisk finisher threads = 3 2024-01-15T11:24:16.026 INFO:teuthology.orchestra.run.smithi141.stdout: filestore apply finisher threads = 3 2024-01-15T11:24:16.027 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.027 INFO:teuthology.orchestra.run.smithi141.stdout: bdev debug aio = true 2024-01-15T11:24:16.027 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug misdirected ops = true 2024-01-15T11:24:16.027 INFO:teuthology.orchestra.run.smithi141.stdout: bdev async discard = True 2024-01-15T11:24:16.027 INFO:teuthology.orchestra.run.smithi141.stdout: bdev enable discard = True 2024-01-15T11:24:16.028 INFO:teuthology.orchestra.run.smithi141.stdout: bluestore allocator = bitmap 2024-01-15T11:24:16.028 INFO:teuthology.orchestra.run.smithi141.stdout: bluestore block size = 96636764160 2024-01-15T11:24:16.028 INFO:teuthology.orchestra.run.smithi141.stdout: bluestore fsck on mount = True 2024-01-15T11:24:16.028 INFO:teuthology.orchestra.run.smithi141.stdout: debug bluefs = 1/20 2024-01-15T11:24:16.028 INFO:teuthology.orchestra.run.smithi141.stdout: debug bluestore = 1/20 2024-01-15T11:24:16.029 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2024-01-15T11:24:16.029 INFO:teuthology.orchestra.run.smithi141.stdout: debug osd = 20 2024-01-15T11:24:16.029 INFO:teuthology.orchestra.run.smithi141.stdout: debug rocksdb = 4/10 2024-01-15T11:24:16.029 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd backfillfull_ratio = 0.85 2024-01-15T11:24:16.029 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd full ratio = 0.9 2024-01-15T11:24:16.029 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd nearfull ratio = 0.8 2024-01-15T11:24:16.030 INFO:teuthology.orchestra.run.smithi141.stdout: osd failsafe full ratio = 0.95 2024-01-15T11:24:16.030 INFO:teuthology.orchestra.run.smithi141.stdout: osd objectstore = bluestore 2024-01-15T11:24:16.030 INFO:teuthology.orchestra.run.smithi141.stdout: osd op complaint time = 180 2024-01-15T11:24:16.030 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.030 INFO:teuthology.orchestra.run.smithi141.stdout:[mgr] 2024-01-15T11:24:16.031 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2024-01-15T11:24:16.031 INFO:teuthology.orchestra.run.smithi141.stdout: debug mgr = 20 2024-01-15T11:24:16.031 INFO:teuthology.orchestra.run.smithi141.stdout: debug mon = 20 2024-01-15T11:24:16.031 INFO:teuthology.orchestra.run.smithi141.stdout: debug auth = 20 2024-01-15T11:24:16.031 INFO:teuthology.orchestra.run.smithi141.stdout: mon reweight min pgs per osd = 4 2024-01-15T11:24:16.032 INFO:teuthology.orchestra.run.smithi141.stdout: mon reweight min bytes per osd = 10 2024-01-15T11:24:16.032 INFO:teuthology.orchestra.run.smithi141.stdout: mgr/telemetry/nag = false 2024-01-15T11:24:16.032 INFO:teuthology.orchestra.run.smithi141.stdout: debug client = 20 2024-01-15T11:24:16.032 INFO:teuthology.orchestra.run.smithi141.stdout: debug finisher = 20 2024-01-15T11:24:16.032 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.032 INFO:teuthology.orchestra.run.smithi141.stdout:[mon] 2024-01-15T11:24:16.033 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2024-01-15T11:24:16.033 INFO:teuthology.orchestra.run.smithi141.stdout: debug mon = 20 2024-01-15T11:24:16.033 INFO:teuthology.orchestra.run.smithi141.stdout: debug paxos = 20 2024-01-15T11:24:16.033 INFO:teuthology.orchestra.run.smithi141.stdout: debug auth = 20 2024-01-15T11:24:16.033 INFO:teuthology.orchestra.run.smithi141.stdout: mon data avail warn = 5 2024-01-15T11:24:16.034 INFO:teuthology.orchestra.run.smithi141.stdout: mon mgr mkfs grace = 240 2024-01-15T11:24:16.034 INFO:teuthology.orchestra.run.smithi141.stdout: mon reweight min pgs per osd = 4 2024-01-15T11:24:16.034 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd reporter subtree level = osd 2024-01-15T11:24:16.034 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd prime pg temp = true 2024-01-15T11:24:16.034 INFO:teuthology.orchestra.run.smithi141.stdout: mon reweight min bytes per osd = 10 2024-01-15T11:24:16.035 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.035 INFO:teuthology.orchestra.run.smithi141.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-15T11:24:16.035 INFO:teuthology.orchestra.run.smithi141.stdout: auth mon ticket ttl = 660 # 11m 2024-01-15T11:24:16.035 INFO:teuthology.orchestra.run.smithi141.stdout: auth service ticket ttl = 240 # 4m 2024-01-15T11:24:16.035 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.035 INFO:teuthology.orchestra.run.smithi141.stdout: # don't complain about insecure global_id in the test suite 2024-01-15T11:24:16.036 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-15T11:24:16.036 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-15T11:24:16.036 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.036 INFO:teuthology.orchestra.run.smithi141.stdout: # 1m isn't quite enough 2024-01-15T11:24:16.036 INFO:teuthology.orchestra.run.smithi141.stdout: mon_down_mkfs_grace = 2m 2024-01-15T11:24:16.037 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.037 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_filestore_osds = false 2024-01-15T11:24:16.037 INFO:teuthology.orchestra.run.smithi141.stdout: mon op complaint time = 120 2024-01-15T11:24:16.037 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-15T11:24:16.037 INFO:teuthology.orchestra.run.smithi141.stdout:[client] 2024-01-15T11:24:16.038 INFO:teuthology.orchestra.run.smithi141.stdout: rgw cache enabled = true 2024-01-15T11:24:16.038 INFO:teuthology.orchestra.run.smithi141.stdout: rgw enable ops log = true 2024-01-15T11:24:16.038 INFO:teuthology.orchestra.run.smithi141.stdout: rgw enable usage log = true 2024-01-15T11:24:16.038 INFO:teuthology.orchestra.run.smithi141.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-15T11:24:16.038 INFO:teuthology.orchestra.run.smithi141.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-15T11:24:16.038 INFO:teuthology.orchestra.run.smithi141.stdout: client mount timeout = 600 2024-01-15T11:24:16.039 INFO:teuthology.orchestra.run.smithi141.stdout: debug client = 20 2024-01-15T11:24:16.039 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2024-01-15T11:24:16.039 INFO:teuthology.orchestra.run.smithi141.stdout: pid file = "" 2024-01-15T11:24:16.039 INFO:teuthology.orchestra.run.smithi141.stdout: rados mon op timeout = 900 2024-01-15T11:24:16.039 INFO:teuthology.orchestra.run.smithi141.stdout: rados osd op timeout = 900 2024-01-15T11:24:16.040 INFO:teuthology.orchestra.run.smithi141.stdout:[mon.a] 2024-01-15T11:24:16.040 INFO:teuthology.orchestra.run.smithi141.stdout:[mon.b] 2024-01-15T11:24:16.040 INFO:teuthology.orchestra.run.smithi141.stdout:[mon.c] 2024-01-15T11:24:16.040 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.a] 2024-01-15T11:24:16.040 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.c] 2024-01-15T11:24:16.041 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.b] 2024-01-15T11:24:16.041 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.d] 2024-01-15T11:24:16.041 INFO:teuthology.orchestra.run.smithi141.stdout:[mds] 2024-01-15T11:24:16.041 INFO:teuthology.orchestra.run.smithi141.stdout: debug mds = 20 2024-01-15T11:24:16.041 INFO:teuthology.orchestra.run.smithi141.stdout: debug mds balancer = 20 2024-01-15T11:24:16.042 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2024-01-15T11:24:16.042 INFO:teuthology.orchestra.run.smithi141.stdout: mds debug frag = True 2024-01-15T11:24:16.042 INFO:teuthology.orchestra.run.smithi141.stdout: mds debug scatterstat = True 2024-01-15T11:24:16.042 INFO:teuthology.orchestra.run.smithi141.stdout: mds op complaint time = 180 2024-01-15T11:24:16.042 INFO:teuthology.orchestra.run.smithi141.stdout: mds verify scatter = True 2024-01-15T11:24:16.043 INFO:teuthology.orchestra.run.smithi141.stdout: osd op complaint time = 180 2024-01-15T11:24:16.043 INFO:teuthology.orchestra.run.smithi141.stdout: rados mon op timeout = 900 2024-01-15T11:24:16.043 INFO:teuthology.orchestra.run.smithi141.stdout: rados osd op timeout = 900 2024-01-15T11:24:16.043 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-15T11:24:16.044 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-15T11:24:16.110 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-15T11:24:16.110 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:16.110 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-15T11:24:16.127 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:16.128 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-15T11:24:16.184 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi113.front.sepia.ceph.com 2024-01-15T11:24:16.184 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:16.185 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-15T11:24:16.185 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-15T11:24:16.261 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:16.262 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:16.319 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi141.front.sepia.ceph.com 2024-01-15T11:24:16.319 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:16.319 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-15T11:24:16.319 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-15T11:24:16.357 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:16.358 DEBUG:teuthology.orchestra.run.smithi141:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:16.416 INFO:tasks.ceph:Setting up mon nodes... 2024-01-15T11:24:16.416 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-15T11:24:16.417 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-15T11:24:16.470 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-15T11:24:16.472 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-15T11:24:16.550 INFO:teuthology.orchestra.run.smithi141.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-15T11:24:16.552 INFO:tasks.ceph:Setting up mds nodes... 2024-01-15T11:24:16.553 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-01-15T11:24:16.606 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-01-15T11:24:16.609 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-01-15T11:24:16.635 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-01-15T11:24:16.730 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-01-15T11:24:16.732 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-01-15T11:24:16.758 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-01-15T11:24:16.817 INFO:teuthology.orchestra.run.smithi141.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-01-15T11:24:16.820 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-01-15T11:24:16.847 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-01-15T11:24:16.942 INFO:teuthology.orchestra.run.smithi141.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-01-15T11:24:16.945 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-01-15T11:24:16.971 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-15T11:24:16.971 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-15T11:24:17.015 INFO:teuthology.orchestra.run.smithi113.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-15T11:24:17.029 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-01-15T11:24:17.112 INFO:teuthology.orchestra.run.smithi113.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-01-15T11:24:17.125 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-15T11:24:17.168 INFO:teuthology.orchestra.run.smithi141.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-15T11:24:17.181 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-01-15T11:24:17.265 INFO:teuthology.orchestra.run.smithi141.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-01-15T11:24:17.278 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-15T11:24:17.278 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi113.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi141.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-15T11:24:17.279 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-15T11:24:17.305 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T11:24:17.305 INFO:tasks.ceph:role: osd.0 2024-01-15T11:24:17.305 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi113.front.sepia.ceph.com 2024-01-15T11:24:17.305 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-15T11:24:17.380 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T11:24:17.380 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T11:24:17.380 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T11:24:17.380 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T11:24:17.380 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T11:24:17.380 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2024-01-15T11:24:17.380 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T11:24:17.381 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T11:24:17.381 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T11:24:17.381 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T11:24:17.861 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2024-01-15T11:24:17.863 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2024-01-15T11:24:17.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-15T11:24:18.003 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-15T11:24:18.037 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-15T11:24:18.104 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T11:24:18.104 INFO:tasks.ceph:role: osd.1 2024-01-15T11:24:18.104 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi113.front.sepia.ceph.com 2024-01-15T11:24:18.104 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-15T11:24:18.178 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T11:24:18.179 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T11:24:18.179 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T11:24:18.179 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T11:24:18.179 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T11:24:18.179 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2024-01-15T11:24:18.179 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T11:24:18.179 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T11:24:18.179 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T11:24:18.179 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T11:24:18.648 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2024-01-15T11:24:18.650 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2024-01-15T11:24:18.651 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-15T11:24:18.772 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-15T11:24:18.806 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-15T11:24:18.873 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T11:24:18.873 INFO:tasks.ceph:role: osd.2 2024-01-15T11:24:18.873 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi113.front.sepia.ceph.com 2024-01-15T11:24:18.873 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-15T11:24:18.947 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T11:24:18.947 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T11:24:18.948 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T11:24:18.948 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T11:24:18.948 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T11:24:18.948 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2024-01-15T11:24:18.948 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T11:24:18.948 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T11:24:18.948 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T11:24:18.948 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T11:24:19.427 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2024-01-15T11:24:19.429 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2024-01-15T11:24:19.429 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-15T11:24:19.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-15T11:24:19.597 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-15T11:24:19.665 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T11:24:19.665 INFO:tasks.ceph:role: osd.3 2024-01-15T11:24:19.665 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi113.front.sepia.ceph.com 2024-01-15T11:24:19.665 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-15T11:24:19.739 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T11:24:19.740 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T11:24:19.740 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T11:24:19.740 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T11:24:19.740 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T11:24:19.740 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2024-01-15T11:24:19.740 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T11:24:19.740 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T11:24:19.740 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T11:24:19.740 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T11:24:20.222 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2024-01-15T11:24:20.223 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2024-01-15T11:24:20.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-15T11:24:20.369 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-15T11:24:20.402 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:20.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:20.492+0000 7f3ca36d3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-15T11:24:20.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:20.494+0000 7f3ca36d3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-15T11:24:20.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:20.494+0000 7f3ca36d3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-15T11:24:20.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:20.494+0000 7f3ca36d3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-15T11:24:20.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:20.494+0000 7f3ca36d3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-15T11:24:20.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:20.496+0000 7f3ca36d3640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-15T11:24:23.343 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-15T11:24:23.374 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:23.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:23.468+0000 7f0ecdbc4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-15T11:24:23.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:23.468+0000 7f0ecdbc4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-15T11:24:23.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:23.468+0000 7f0ecdbc4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-15T11:24:23.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:23.468+0000 7f0ecdbc4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-15T11:24:23.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:23.468+0000 7f0ecdbc4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-15T11:24:23.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:23.468+0000 7f0ecdbc4640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-15T11:24:26.320 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-15T11:24:26.352 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:26.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:26.448+0000 7fbfc4e3c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-15T11:24:26.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:26.448+0000 7fbfc4e3c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-15T11:24:26.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:26.448+0000 7fbfc4e3c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-15T11:24:26.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:26.448+0000 7fbfc4e3c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-15T11:24:26.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:26.448+0000 7fbfc4e3c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-15T11:24:26.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:26.448+0000 7fbfc4e3c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-15T11:24:29.316 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-15T11:24:29.347 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:29.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:29.442+0000 7f62dce1a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-15T11:24:29.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:29.443+0000 7f62dce1a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-15T11:24:29.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:29.443+0000 7f62dce1a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-15T11:24:29.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:29.443+0000 7f62dce1a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-15T11:24:29.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:29.443+0000 7f62dce1a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-15T11:24:29.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:29.443+0000 7f62dce1a640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-15T11:24:32.294 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-15T11:24:32.327 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-15T11:24:32.362 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T11:24:32.362 INFO:tasks.ceph:role: osd.4 2024-01-15T11:24:32.362 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi141.front.sepia.ceph.com 2024-01-15T11:24:32.362 DEBUG:teuthology.orchestra.run.smithi141:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-15T11:24:32.440 INFO:teuthology.orchestra.run.smithi141.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T11:24:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T11:24:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T11:24:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T11:24:32.441 INFO:teuthology.orchestra.run.smithi141.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T11:24:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: = sunit=0 swidth=0 blks 2024-01-15T11:24:32.441 INFO:teuthology.orchestra.run.smithi141.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T11:24:32.441 INFO:teuthology.orchestra.run.smithi141.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T11:24:32.441 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T11:24:32.441 INFO:teuthology.orchestra.run.smithi141.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T11:24:32.879 INFO:teuthology.orchestra.run.smithi141.stdout:Discarding blocks...Done. 2024-01-15T11:24:32.881 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi141.front.sepia.ceph.com -o noatime 2024-01-15T11:24:32.881 DEBUG:teuthology.orchestra.run.smithi141:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-15T11:24:33.062 DEBUG:teuthology.orchestra.run.smithi141:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-15T11:24:33.099 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-15T11:24:33.169 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T11:24:33.169 INFO:tasks.ceph:role: osd.5 2024-01-15T11:24:33.169 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi141.front.sepia.ceph.com 2024-01-15T11:24:33.169 DEBUG:teuthology.orchestra.run.smithi141:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-15T11:24:33.244 INFO:teuthology.orchestra.run.smithi141.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T11:24:33.244 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T11:24:33.244 INFO:teuthology.orchestra.run.smithi141.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T11:24:33.244 INFO:teuthology.orchestra.run.smithi141.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T11:24:33.245 INFO:teuthology.orchestra.run.smithi141.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T11:24:33.245 INFO:teuthology.orchestra.run.smithi141.stdout: = sunit=0 swidth=0 blks 2024-01-15T11:24:33.245 INFO:teuthology.orchestra.run.smithi141.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T11:24:33.245 INFO:teuthology.orchestra.run.smithi141.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T11:24:33.245 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T11:24:33.245 INFO:teuthology.orchestra.run.smithi141.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T11:24:33.670 INFO:teuthology.orchestra.run.smithi141.stdout:Discarding blocks...Done. 2024-01-15T11:24:33.671 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi141.front.sepia.ceph.com -o noatime 2024-01-15T11:24:33.672 DEBUG:teuthology.orchestra.run.smithi141:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-15T11:24:33.778 DEBUG:teuthology.orchestra.run.smithi141:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-15T11:24:33.811 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-15T11:24:33.880 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T11:24:33.881 INFO:tasks.ceph:role: osd.6 2024-01-15T11:24:33.881 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi141.front.sepia.ceph.com 2024-01-15T11:24:33.881 DEBUG:teuthology.orchestra.run.smithi141:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-15T11:24:33.956 INFO:teuthology.orchestra.run.smithi141.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T11:24:33.956 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T11:24:33.956 INFO:teuthology.orchestra.run.smithi141.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T11:24:33.956 INFO:teuthology.orchestra.run.smithi141.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T11:24:33.957 INFO:teuthology.orchestra.run.smithi141.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T11:24:33.957 INFO:teuthology.orchestra.run.smithi141.stdout: = sunit=0 swidth=0 blks 2024-01-15T11:24:33.957 INFO:teuthology.orchestra.run.smithi141.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T11:24:33.957 INFO:teuthology.orchestra.run.smithi141.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T11:24:33.957 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T11:24:33.957 INFO:teuthology.orchestra.run.smithi141.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T11:24:34.412 INFO:teuthology.orchestra.run.smithi141.stdout:Discarding blocks...Done. 2024-01-15T11:24:34.414 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi141.front.sepia.ceph.com -o noatime 2024-01-15T11:24:34.414 DEBUG:teuthology.orchestra.run.smithi141:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-15T11:24:34.595 DEBUG:teuthology.orchestra.run.smithi141:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-15T11:24:34.630 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-15T11:24:34.700 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T11:24:34.700 INFO:tasks.ceph:role: osd.7 2024-01-15T11:24:34.700 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi141.front.sepia.ceph.com 2024-01-15T11:24:34.701 DEBUG:teuthology.orchestra.run.smithi141:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-15T11:24:34.777 INFO:teuthology.orchestra.run.smithi141.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T11:24:34.777 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T11:24:34.777 INFO:teuthology.orchestra.run.smithi141.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T11:24:34.777 INFO:teuthology.orchestra.run.smithi141.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T11:24:34.778 INFO:teuthology.orchestra.run.smithi141.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T11:24:34.778 INFO:teuthology.orchestra.run.smithi141.stdout: = sunit=0 swidth=0 blks 2024-01-15T11:24:34.778 INFO:teuthology.orchestra.run.smithi141.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T11:24:34.778 INFO:teuthology.orchestra.run.smithi141.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T11:24:34.778 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T11:24:34.778 INFO:teuthology.orchestra.run.smithi141.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T11:24:35.217 INFO:teuthology.orchestra.run.smithi141.stdout:Discarding blocks...Done. 2024-01-15T11:24:35.219 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi141.front.sepia.ceph.com -o noatime 2024-01-15T11:24:35.219 DEBUG:teuthology.orchestra.run.smithi141:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-15T11:24:35.381 DEBUG:teuthology.orchestra.run.smithi141:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-15T11:24:35.417 DEBUG:teuthology.orchestra.run.smithi141:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:35.509+0000 7fa66f178640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-15T11:24:35.507 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:35.509+0000 7fa66f178640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-15T11:24:35.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:35.509+0000 7fa66f178640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-15T11:24:35.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:35.509+0000 7fa66f178640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-15T11:24:35.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:35.509+0000 7fa66f178640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-15T11:24:35.508 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:35.510+0000 7fa66f178640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-15T11:24:38.361 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-15T11:24:38.393 DEBUG:teuthology.orchestra.run.smithi141:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:38.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:38.487+0000 7f115cab6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-15T11:24:38.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:38.488+0000 7f115cab6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-15T11:24:38.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:38.488+0000 7f115cab6640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-15T11:24:38.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:38.488+0000 7f115cab6640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-15T11:24:38.486 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:38.488+0000 7f115cab6640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-15T11:24:38.487 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:38.488+0000 7f115cab6640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-15T11:24:41.340 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-15T11:24:41.372 DEBUG:teuthology.orchestra.run.smithi141:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:41.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:41.465+0000 7fc2a066c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-15T11:24:41.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:41.465+0000 7fc2a066c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-15T11:24:41.463 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:41.465+0000 7fc2a066c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-15T11:24:41.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:41.465+0000 7fc2a066c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-15T11:24:41.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:41.465+0000 7fc2a066c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-15T11:24:41.464 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:41.466+0000 7fc2a066c640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-15T11:24:43.926 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:24:43.929 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:24:44.324 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-15T11:24:44.357 DEBUG:teuthology.orchestra.run.smithi141:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:44.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:44.451+0000 7f5b5ecab640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-15T11:24:44.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:44.451+0000 7f5b5ecab640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-15T11:24:44.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:44.451+0000 7f5b5ecab640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-15T11:24:44.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:44.451+0000 7f5b5ecab640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-15T11:24:44.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:44.451+0000 7f5b5ecab640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-15T11:24:44.450 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:44.452+0000 7f5b5ecab640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-15T11:24:47.299 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-15T11:24:47.331 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-15T11:24:47.331 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:47.332 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-15T11:24:47.364 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:47.364 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-01-15T11:24:47.435 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:47.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-01-15T11:24:47.504 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:47.505 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-15T11:24:47.574 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:47.576 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-15T11:24:47.642 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:47.642 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-15T11:24:47.710 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:47.710 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-15T11:24:47.777 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:47.777 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-15T11:24:47.810 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:47.810 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-01-15T11:24:47.883 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:47.883 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-01-15T11:24:47.954 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:47.955 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-15T11:24:48.026 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:48.026 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-15T11:24:48.096 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:48.096 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-15T11:24:48.166 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:48.166 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-15T11:24:48.235 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:48.235 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-15T11:24:48.253 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:48.254 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-01-15T11:24:48.312 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:48.312 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-15T11:24:48.331 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:48.331 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-01-15T11:24:48.390 INFO:tasks.ceph:Adding keys to all mons... 2024-01-15T11:24:48.390 DEBUG:teuthology.orchestra.run.smithi113:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-15T11:24:48.393 DEBUG:teuthology.orchestra.run.smithi141:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-15T11:24:48.434 INFO:teuthology.orchestra.run.smithi113.stdout:[mgr.y] 2024-01-15T11:24:48.434 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDgFaVl/742HBAAGNipQFrMT9muRnaRJ8ew1A== 2024-01-15T11:24:48.434 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.a] 2024-01-15T11:24:48.434 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDgFaVl3qtfJBAApmUPtVUchbLyUR5i7tMBcQ== 2024-01-15T11:24:48.434 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.c] 2024-01-15T11:24:48.434 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDgFaVlYja2KxAAVvtunMrSignFsgv9+m6zqw== 2024-01-15T11:24:48.434 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.0] 2024-01-15T11:24:48.435 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDkFaVlVLNkHRAAsJe9+2Vra6oX5NYS69d1IQ== 2024-01-15T11:24:48.435 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.1] 2024-01-15T11:24:48.435 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDnFaVlQ5v2GxAADh/Lh54BWAWEqoBa7khCZQ== 2024-01-15T11:24:48.435 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.2] 2024-01-15T11:24:48.435 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDqFaVl2z7GGhAAwW5bYhZYJdr+UX0VaFseSg== 2024-01-15T11:24:48.435 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.3] 2024-01-15T11:24:48.435 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDtFaVl6/ZmGhAAhk5GCH8FAsH8mGU/6BjG0w== 2024-01-15T11:24:48.435 INFO:teuthology.orchestra.run.smithi113.stdout:[mgr.x] 2024-01-15T11:24:48.435 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDgFaVl0fP/IBAAAs2u2ra8ZJ6cSemJuZi/Ww== 2024-01-15T11:24:48.435 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.b] 2024-01-15T11:24:48.436 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDgFaVll1DvMBAAyj6YKcqFWrYiuWCnEzfZyQ== 2024-01-15T11:24:48.436 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.d] 2024-01-15T11:24:48.436 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDgFaVl4jdhOBAAHlJN1TpsKN9lU8hjxWW89w== 2024-01-15T11:24:48.436 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.4] 2024-01-15T11:24:48.436 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDzFaVldKVuHhAAQV5AFYI6jisYR5QFMqpUUg== 2024-01-15T11:24:48.436 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.5] 2024-01-15T11:24:48.436 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD2FaVl7KklHRAA2VXFQIclhfkuliHmpWaK4g== 2024-01-15T11:24:48.436 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.6] 2024-01-15T11:24:48.436 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD5FaVl9m3NGxAAfiz7aeLWympZqVOfV19K4Q== 2024-01-15T11:24:48.436 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.7] 2024-01-15T11:24:48.437 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD8FaVlKbH8GhAAf+GPFNeLp2+s3jTtqklCmw== 2024-01-15T11:24:48.437 INFO:teuthology.orchestra.run.smithi113.stdout:[client.0] 2024-01-15T11:24:48.437 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDhFaVl1ysgARAAgjFqY3kbPDbaQ0A6WxgMLQ== 2024-01-15T11:24:48.437 INFO:teuthology.orchestra.run.smithi113.stdout:[client.2] 2024-01-15T11:24:48.437 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDhFaVlHvXmBhAAttmTrSddVurihd5SlwQhVw== 2024-01-15T11:24:48.437 INFO:teuthology.orchestra.run.smithi113.stdout:[client.1] 2024-01-15T11:24:48.437 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDhFaVlS0RDChAAfGRj4v0fLzIClr6FnNmemw== 2024-01-15T11:24:48.437 INFO:teuthology.orchestra.run.smithi113.stdout:[client.3] 2024-01-15T11:24:48.437 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDhFaVluyUHEBAA26iiKREXkJTEjQePzxY4JQ== 2024-01-15T11:24:48.457 INFO:teuthology.orchestra.run.smithi141.stdout:[mgr.y] 2024-01-15T11:24:48.457 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDgFaVl/742HBAAGNipQFrMT9muRnaRJ8ew1A== 2024-01-15T11:24:48.457 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.a] 2024-01-15T11:24:48.457 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDgFaVl3qtfJBAApmUPtVUchbLyUR5i7tMBcQ== 2024-01-15T11:24:48.457 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.c] 2024-01-15T11:24:48.457 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDgFaVlYja2KxAAVvtunMrSignFsgv9+m6zqw== 2024-01-15T11:24:48.458 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.0] 2024-01-15T11:24:48.458 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDkFaVlVLNkHRAAsJe9+2Vra6oX5NYS69d1IQ== 2024-01-15T11:24:48.458 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.1] 2024-01-15T11:24:48.458 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDnFaVlQ5v2GxAADh/Lh54BWAWEqoBa7khCZQ== 2024-01-15T11:24:48.458 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.2] 2024-01-15T11:24:48.458 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDqFaVl2z7GGhAAwW5bYhZYJdr+UX0VaFseSg== 2024-01-15T11:24:48.458 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.3] 2024-01-15T11:24:48.458 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDtFaVl6/ZmGhAAhk5GCH8FAsH8mGU/6BjG0w== 2024-01-15T11:24:48.458 INFO:teuthology.orchestra.run.smithi141.stdout:[mgr.x] 2024-01-15T11:24:48.458 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDgFaVl0fP/IBAAAs2u2ra8ZJ6cSemJuZi/Ww== 2024-01-15T11:24:48.458 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.b] 2024-01-15T11:24:48.459 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDgFaVll1DvMBAAyj6YKcqFWrYiuWCnEzfZyQ== 2024-01-15T11:24:48.459 INFO:teuthology.orchestra.run.smithi141.stdout:[mds.d] 2024-01-15T11:24:48.459 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDgFaVl4jdhOBAAHlJN1TpsKN9lU8hjxWW89w== 2024-01-15T11:24:48.459 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.4] 2024-01-15T11:24:48.459 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDzFaVldKVuHhAAQV5AFYI6jisYR5QFMqpUUg== 2024-01-15T11:24:48.459 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.5] 2024-01-15T11:24:48.459 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQD2FaVl7KklHRAA2VXFQIclhfkuliHmpWaK4g== 2024-01-15T11:24:48.459 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.6] 2024-01-15T11:24:48.459 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQD5FaVl9m3NGxAAfiz7aeLWympZqVOfV19K4Q== 2024-01-15T11:24:48.459 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.7] 2024-01-15T11:24:48.459 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQD8FaVlKbH8GhAAf+GPFNeLp2+s3jTtqklCmw== 2024-01-15T11:24:48.460 INFO:teuthology.orchestra.run.smithi141.stdout:[client.0] 2024-01-15T11:24:48.460 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDhFaVl1ysgARAAgjFqY3kbPDbaQ0A6WxgMLQ== 2024-01-15T11:24:48.460 INFO:teuthology.orchestra.run.smithi141.stdout:[client.2] 2024-01-15T11:24:48.460 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDhFaVlHvXmBhAAttmTrSddVurihd5SlwQhVw== 2024-01-15T11:24:48.460 INFO:teuthology.orchestra.run.smithi141.stdout:[client.1] 2024-01-15T11:24:48.460 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDhFaVlS0RDChAAfGRj4v0fLzIClr6FnNmemw== 2024-01-15T11:24:48.460 INFO:teuthology.orchestra.run.smithi141.stdout:[client.3] 2024-01-15T11:24:48.460 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQDhFaVluyUHEBAA26iiKREXkJTEjQePzxY4JQ== 2024-01-15T11:24:48.461 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T11:24:48.482 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T11:24:48.560 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T11:24:48.562 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T11:24:48.620 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T11:24:48.623 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T11:24:48.679 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:48.682 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:48.736 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:48.739 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:48.793 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:48.795 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:48.850 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:48.852 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:48.906 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T11:24:48.908 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T11:24:48.962 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T11:24:48.964 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T11:24:49.021 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T11:24:49.024 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T11:24:49.082 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:49.085 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:49.139 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:49.142 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:49.196 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:49.198 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:49.253 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:49.255 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T11:24:49.309 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T11:24:49.312 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T11:24:49.369 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T11:24:49.372 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T11:24:49.429 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T11:24:49.431 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T11:24:49.489 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T11:24:49.492 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T11:24:49.548 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-15T11:24:49.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-15T11:24:49.576 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-15T11:24:49.674 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-15T11:24:49.700 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-15T11:24:49.727 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-15T11:24:49.825 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-15T11:24:49.852 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-15T11:24:49.918 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-15T11:24:50.018 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-15T11:24:50.045 DEBUG:teuthology.orchestra.run.smithi113:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:50.047 DEBUG:teuthology.orchestra.run.smithi141:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-15T11:24:50.102 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-15T11:24:50.102 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-15T11:24:50.102 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-15T11:24:50.104 INFO:tasks.ceph.mon.a:Started 2024-01-15T11:24:50.105 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-15T11:24:50.105 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-01-15T11:24:50.144 INFO:tasks.ceph.mon.b:Started 2024-01-15T11:24:50.144 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-15T11:24:50.145 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-01-15T11:24:50.147 INFO:tasks.ceph.mon.c:Started 2024-01-15T11:24:50.147 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-15T11:24:50.147 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-15T11:24:50.147 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-01-15T11:24:50.149 INFO:tasks.ceph.mgr.y:Started 2024-01-15T11:24:50.150 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-15T11:24:50.150 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-15T11:24:50.152 INFO:tasks.ceph.mgr.x:Started 2024-01-15T11:24:50.152 INFO:tasks.ceph:Setting crush tunables to default 2024-01-15T11:24:50.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd crush tunables default 2024-01-15T11:24:50.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.278+0000 7f038f90d700 1 Processor -- start 2024-01-15T11:24:50.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.278+0000 7f038f90d700 1 -- start start 2024-01-15T11:24:50.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.279+0000 7f038f90d700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0388161560 0x7f03880b6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:50.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.279+0000 7f038f90d700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0388161940 con 0x7f03880b9ba0 2024-01-15T11:24:50.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.279+0000 7f038f90d700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03880b7940 con 0x7f03880b7410 2024-01-15T11:24:50.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.279+0000 7f038f90d700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03880b85c0 con 0x7f0388161560 2024-01-15T11:24:50.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.279+0000 7f038cea8700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f03880b9ba0 0x7f03880d1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:49220/0 (socket says 172.21.15.113:49220) 2024-01-15T11:24:50.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.279+0000 7f038cea8700 1 -- 172.21.15.113:0/2615454719 learned_addr learned my addr 172.21.15.113:0/2615454719 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:24:50.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.279+0000 7f038d6a9700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0388161560 0x7f03880b6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:50.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.279+0000 7f038d6a9700 1 -- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0388161560 msgr2=0x7f03880b6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-15T11:24:50.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.279+0000 7f038d6a9700 1 -- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0388161560 msgr2=0x7f03880b6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-15T11:24:50.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.279+0000 7f038d6a9700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0388161560 0x7f03880b6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-15T11:24:50.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.279+0000 7f038d6a9700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0388161560 0x7f03880b6ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-15T11:24:50.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.457+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843193192 0 0) 0x7f0388161940 con 0x7f03880b9ba0 2024-01-15T11:24:50.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03740035b0 con 0x7f03880b9ba0 2024-01-15T11:24:50.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055634174 0 0) 0x7f03880b7940 con 0x7f03880b7410 2024-01-15T11:24:50.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0388161940 con 0x7f03880b7410 2024-01-15T11:24:50.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 <== mon.1 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 812734408 0 0) 0x7f0388161940 con 0x7f03880b7410 2024-01-15T11:24:50.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0388161560 msgr2=0x7f03880b6ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T11:24:50.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0388161560 0x7f03880b6ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:50.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 >> v1:172.21.15.113:6789/0 conn(0x7f03880b9ba0 legacy=0x7f03880d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:50.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03880b7c20 con 0x7f03880b7410 2024-01-15T11:24:50.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 <== mon.1 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f03800019f0 con 0x7f03880b7410 2024-01-15T11:24:50.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 >> v1:172.21.15.141:6789/0 conn(0x7f03880b7410 legacy=0x7f03880b89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:50.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0374003b90 0x7f0374003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:50.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0388161560 0x7f0374004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:50.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0374004d90 0x7f0374005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:50.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880b85c0 con 0x7f0374003b90 2024-01-15T11:24:50.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037400bba0 con 0x7f0374004d90 2024-01-15T11:24:50.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.458+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037400bd30 con 0x7f0388161560 2024-01-15T11:24:50.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f038d6a9700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0374003b90 0x7f0374003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:50.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f038deaa700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0374004d90 0x7f0374005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:50.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f038cea8700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0388161560 0x7f0374004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:50.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f038d6a9700 1 -- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0388161560 msgr2=0x7f0374004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:24:50.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f038d6a9700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0388161560 0x7f0374004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:50.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f038d6a9700 1 -- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0374004d90 msgr2=0x7f0374005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:24:50.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f038d6a9700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0374004d90 0x7f0374005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:50.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f038d6a9700 1 -- 172.21.15.113:0/2615454719 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0388161940 con 0x7f0374003b90 2024-01-15T11:24:50.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f038deaa700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0374004d90 0x7f0374005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:24:50.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f038d6a9700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0374003b90 0x7f0374003f90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f03780098c0 tx=0x7f0374005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d39d1a610fecac97 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:50.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037800c070 con 0x7f0374003b90 2024-01-15T11:24:50.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0378004020 con 0x7f0374003b90 2024-01-15T11:24:50.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.459+0000 7f037ffff700 1 -- 172.21.15.113:0/2615454719 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03780043d0 con 0x7f0374003b90 2024-01-15T11:24:50.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.460+0000 7f038f90d700 1 -- 172.21.15.113:0/2615454719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0374003b90 msgr2=0x7f0374003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:50.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.460+0000 7f038f90d700 1 --2- 172.21.15.113:0/2615454719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0374003b90 0x7f0374003f90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f03780098c0 tx=0x7f0374005880 comp rx=0 tx=0).stop 2024-01-15T11:24:50.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.460+0000 7f038deaa700 1 -- 172.21.15.113:0/2615454719 reap_dead start 2024-01-15T11:24:50.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.460+0000 7f038f90d700 1 -- 172.21.15.113:0/2615454719 shutdown_connections 2024-01-15T11:24:50.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.460+0000 7f038f90d700 1 -- 172.21.15.113:0/2615454719 >> 172.21.15.113:0/2615454719 conn(0x7f038806b410 msgr2=0x7f038806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:50.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.460+0000 7f038f90d700 1 -- 172.21.15.113:0/2615454719 shutdown_connections 2024-01-15T11:24:50.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.460+0000 7f038f90d700 1 -- 172.21.15.113:0/2615454719 wait complete. 2024-01-15T11:24:50.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.462+0000 7f038f90d700 1 Processor -- start 2024-01-15T11:24:50.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.462+0000 7f038f90d700 1 -- start start 2024-01-15T11:24:50.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038f90d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0374003b90 0x7f03880acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:50.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038f90d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f03880cce70 0x7f03880c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:50.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038f90d700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f03880c9d30 0x7f03881f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:50.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038f90d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880ca1c0 con 0x7f0374003b90 2024-01-15T11:24:50.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038f90d700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880cc9e0 con 0x7f03880cce70 2024-01-15T11:24:50.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038f90d700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03881f4860 con 0x7f03880c9d30 2024-01-15T11:24:50.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038d6a9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0374003b90 0x7f03880acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:50.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038d6a9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0374003b90 0x7f03880acfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42226/0 (socket says 172.21.15.113:42226) 2024-01-15T11:24:50.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038d6a9700 1 -- 172.21.15.113:0/2272124850 learned_addr learned my addr 172.21.15.113:0/2272124850 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:24:50.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038deaa700 1 --2- 172.21.15.113:0/2272124850 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f03880c9d30 0x7f03881f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:50.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038cea8700 1 --2- 172.21.15.113:0/2272124850 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f03880cce70 0x7f03880c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:50.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038d6a9700 1 -- 172.21.15.113:0/2272124850 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f03880c9d30 msgr2=0x7f03881f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:50.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038d6a9700 1 --2- 172.21.15.113:0/2272124850 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f03880c9d30 0x7f03881f3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:50.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038d6a9700 1 -- 172.21.15.113:0/2272124850 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f03880cce70 msgr2=0x7f03880c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:50.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038d6a9700 1 --2- 172.21.15.113:0/2272124850 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f03880cce70 0x7f03880c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:50.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038d6a9700 1 -- 172.21.15.113:0/2272124850 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03880cc0b0 con 0x7f0374003b90 2024-01-15T11:24:50.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.463+0000 7f038d6a9700 1 --2- 172.21.15.113:0/2272124850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0374003b90 0x7f03880acfb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f03740057f0 tx=0x7f03780047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:50.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.466+0000 7f037dffb700 1 -- 172.21.15.113:0/2272124850 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03780045c0 con 0x7f0374003b90 2024-01-15T11:24:50.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.466+0000 7f037dffb700 1 -- 172.21.15.113:0/2272124850 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0378021070 con 0x7f0374003b90 2024-01-15T11:24:50.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.466+0000 7f037dffb700 1 -- 172.21.15.113:0/2272124850 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037801d450 con 0x7f0374003b90 2024-01-15T11:24:50.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.466+0000 7f038f90d700 1 -- 172.21.15.113:0/2272124850 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03880cc390 con 0x7f0374003b90 2024-01-15T11:24:50.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.466+0000 7f038f90d700 1 -- 172.21.15.113:0/2272124850 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03881f4000 con 0x7f0374003b90 2024-01-15T11:24:50.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.466+0000 7f037dffb700 1 -- 172.21.15.113:0/2272124850 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f037801d5b0 con 0x7f0374003b90 2024-01-15T11:24:50.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.466+0000 7f037dffb700 1 -- 172.21.15.113:0/2272124850 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0378023070 con 0x7f0374003b90 2024-01-15T11:24:50.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.467+0000 7f038f90d700 1 -- 172.21.15.113:0/2272124850 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03880cccc0 con 0x7f0374003b90 2024-01-15T11:24:50.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.469+0000 7f037dffb700 1 -- 172.21.15.113:0/2272124850 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f037802c020 con 0x7f0374003b90 2024-01-15T11:24:50.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.544+0000 7f038f90d700 1 -- 172.21.15.113:0/2272124850 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0388069020 con 0x7f0374003b90 2024-01-15T11:24:50.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.546+0000 7f037dffb700 1 -- 172.21.15.113:0/2272124850 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0378017030 con 0x7f0374003b90 2024-01-15T11:24:50.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.547+0000 7f038f90d700 1 -- 172.21.15.113:0/2272124850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0374003b90 msgr2=0x7f03880acfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:50.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.547+0000 7f038f90d700 1 --2- 172.21.15.113:0/2272124850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0374003b90 0x7f03880acfb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f03740057f0 tx=0x7f03780047c0 comp rx=0 tx=0).stop 2024-01-15T11:24:50.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.547+0000 7f038f90d700 1 -- 172.21.15.113:0/2272124850 shutdown_connections 2024-01-15T11:24:50.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.547+0000 7f038f90d700 1 --2- 172.21.15.113:0/2272124850 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f03880c9d30 0x7f03881f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:50.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.547+0000 7f038f90d700 1 --2- 172.21.15.113:0/2272124850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0374003b90 0x7f03880acfb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:50.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.547+0000 7f038f90d700 1 --2- 172.21.15.113:0/2272124850 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f03880cce70 0x7f03880c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:50.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.547+0000 7f038f90d700 1 -- 172.21.15.113:0/2272124850 >> 172.21.15.113:0/2272124850 conn(0x7f038806b410 msgr2=0x7f03880ce490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:50.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.547+0000 7f038f90d700 1 -- 172.21.15.113:0/2272124850 shutdown_connections 2024-01-15T11:24:50.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:50.547+0000 7f038f90d700 1 -- 172.21.15.113:0/2272124850 wait complete. 2024-01-15T11:24:50.547 INFO:teuthology.orchestra.run.smithi113.stderr:adjusted tunables profile to default 2024-01-15T11:24:50.558 INFO:tasks.ceph:check_enable_crimson: False 2024-01-15T11:24:50.558 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-15T11:24:50.558 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:50.558 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-15T11:24:50.585 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:50.585 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-15T11:24:50.652 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:50.652 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-15T11:24:50.719 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:24:50.719 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-15T11:24:50.787 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:50.788 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-15T11:24:50.815 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:50.815 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-15T11:24:50.872 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:50.875+0000 7f1eb7267200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-15T11:24:50.882 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:50.882 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-15T11:24:50.944 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:50.946+0000 7fa553a50200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-15T11:24:50.948 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:24:50.949 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-15T11:24:50.956 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:50.959+0000 7f1eb7267200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-15T11:24:50.982 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new 04532dcb-3e36-440a-a9c1-c0d2158ab5e0 0 2024-01-15T11:24:51.023 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:51.025+0000 7fa553a50200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-15T11:24:51.033 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:51.036+0000 7f1eb7267200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-15T11:24:51.095 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:51.097+0000 7fa553a50200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-15T11:24:51.107 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:51.110+0000 7f1eb7267200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-15T11:24:51.145 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.148+0000 7f9e16475700 1 Processor -- start 2024-01-15T11:24:51.146 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.148+0000 7f9e16475700 1 -- start start 2024-01-15T11:24:51.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e16475700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9e1015f8d0 0x7f9e100b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e16475700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 0x7f9e100ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e16475700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e100b3860 con 0x7f9e100cffc0 2024-01-15T11:24:51.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e16475700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e100b2520 con 0x7f9e1015f8d0 2024-01-15T11:24:51.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e16475700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e100b2800 con 0x7f9e100d03a0 2024-01-15T11:24:51.153 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e07fff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9e1015f8d0 0x7f9e100b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e14a12700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 0x7f9e100ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e07fff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9e1015f8d0 0x7f9e100b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35842/0 (socket says 172.21.15.141:35842) 2024-01-15T11:24:51.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e07fff700 1 -- 172.21.15.141:0/1850089614 learned_addr learned my addr 172.21.15.141:0/1850089614 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-15T11:24:51.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e14a12700 1 -- 172.21.15.141:0/1850089614 >> v2:172.21.15.141:3300/0 conn(0x7f9e1015f8d0 msgr2=0x7f9e100b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:24:51.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e14a12700 1 --2- 172.21.15.141:0/1850089614 >> v2:172.21.15.141:3300/0 conn(0x7f9e1015f8d0 0x7f9e100b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e14a12700 1 -- 172.21.15.141:0/1850089614 >> v1:172.21.15.141:6789/0 conn(0x7f9e100cffc0 legacy=0x7f9e1015f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e14a12700 1 -- 172.21.15.141:0/1850089614 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e100b2ae0 con 0x7f9e100d03a0 2024-01-15T11:24:51.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e0f7fe700 1 -- 172.21.15.141:0/1850089614 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723569902 0 0) 0x7f9e00001110 con 0x7f9e100cffc0 2024-01-15T11:24:51.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e07fff700 1 --2- 172.21.15.141:0/1850089614 >> v2:172.21.15.141:3300/0 conn(0x7f9e1015f8d0 0x7f9e100b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:51.154 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.149+0000 7f9e14a12700 1 --2- 172.21.15.141:0/1850089614 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 0x7f9e100ccd10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009d90 tx=0x7f9e0800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1574f4170cb6ab4f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:51.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.150+0000 7f9e0f7fe700 1 -- 172.21.15.141:0/1850089614 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0800e070 con 0x7f9e100d03a0 2024-01-15T11:24:51.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.150+0000 7f9e0f7fe700 1 -- 172.21.15.141:0/1850089614 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0800bb20 con 0x7f9e100d03a0 2024-01-15T11:24:51.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.150+0000 7f9e0f7fe700 1 -- 172.21.15.141:0/1850089614 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e08012430 con 0x7f9e100d03a0 2024-01-15T11:24:51.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.150+0000 7f9e16475700 1 -- 172.21.15.141:0/1850089614 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 msgr2=0x7f9e100ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.150+0000 7f9e16475700 1 --2- 172.21.15.141:0/1850089614 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 0x7f9e100ccd10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009d90 tx=0x7f9e0800b330 comp rx=0 tx=0).stop 2024-01-15T11:24:51.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.150+0000 7f9e16475700 1 -- 172.21.15.141:0/1850089614 shutdown_connections 2024-01-15T11:24:51.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.150+0000 7f9e16475700 1 --2- 172.21.15.141:0/1850089614 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 0x7f9e100ccd10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.150+0000 7f9e16475700 1 --2- 172.21.15.141:0/1850089614 >> v2:172.21.15.141:3300/0 conn(0x7f9e1015f8d0 0x7f9e100b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.150+0000 7f9e16475700 1 -- 172.21.15.141:0/1850089614 >> 172.21.15.141:0/1850089614 conn(0x7f9e100d1e50 msgr2=0x7f9e100d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:51.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.150+0000 7f9e16475700 1 -- 172.21.15.141:0/1850089614 shutdown_connections 2024-01-15T11:24:51.155 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.150+0000 7f9e16475700 1 -- 172.21.15.141:0/1850089614 wait complete. 2024-01-15T11:24:51.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.151+0000 7f9e16475700 1 Processor -- start 2024-01-15T11:24:51.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.151+0000 7f9e16475700 1 -- start start 2024-01-15T11:24:51.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.152+0000 7f9e16475700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e100cffc0 0x7f9e101f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.152+0000 7f9e16475700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 0x7f9e1014e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.152+0000 7f9e16475700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1015f8d0 0x7f9e100b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.152+0000 7f9e16475700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e101f18b0 con 0x7f9e1015f8d0 2024-01-15T11:24:51.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.152+0000 7f9e16475700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e100ce4d0 con 0x7f9e100cffc0 2024-01-15T11:24:51.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.152+0000 7f9e16475700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1014ecf0 con 0x7f9e100d03a0 2024-01-15T11:24:51.156 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.152+0000 7f9e07fff700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 0x7f9e1014e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.152+0000 7f9e0ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e100cffc0 0x7f9e101f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.152+0000 7f9e07fff700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 0x7f9e1014e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.141:33676/0 (socket says 172.21.15.141:33676) 2024-01-15T11:24:51.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.152+0000 7f9e0ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e100cffc0 0x7f9e101f1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35852/0 (socket says 172.21.15.141:35852) 2024-01-15T11:24:51.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.152+0000 7f9e07fff700 1 -- 172.21.15.141:0/2766075208 learned_addr learned my addr 172.21.15.141:0/2766075208 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-15T11:24:51.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.152+0000 7f9e14a12700 1 --2- 172.21.15.141:0/2766075208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1015f8d0 0x7f9e100b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e07fff700 1 -- 172.21.15.141:0/2766075208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e100cffc0 msgr2=0x7f9e101f1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e07fff700 1 --2- 172.21.15.141:0/2766075208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e100cffc0 0x7f9e101f1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e07fff700 1 -- 172.21.15.141:0/2766075208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1015f8d0 msgr2=0x7f9e100b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e07fff700 1 --2- 172.21.15.141:0/2766075208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1015f8d0 0x7f9e100b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.157 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e07fff700 1 -- 172.21.15.141:0/2766075208 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df8009550 con 0x7f9e100d03a0 2024-01-15T11:24:51.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e0ffff700 1 --2- 172.21.15.141:0/2766075208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e100cffc0 0x7f9e101f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:51.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e14a12700 1 --2- 172.21.15.141:0/2766075208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1015f8d0 0x7f9e100b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:24:51.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e07fff700 1 --2- 172.21.15.141:0/2766075208 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 0x7f9e1014e7b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9df800e9b0 tx=0x7f9df800ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:51.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e0d7fa700 1 -- 172.21.15.141:0/2766075208 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df800c850 con 0x7f9e100d03a0 2024-01-15T11:24:51.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e0d7fa700 1 -- 172.21.15.141:0/2766075208 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df800c9b0 con 0x7f9e100d03a0 2024-01-15T11:24:51.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e16475700 1 -- 172.21.15.141:0/2766075208 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e08009a70 con 0x7f9e100d03a0 2024-01-15T11:24:51.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e0d7fa700 1 -- 172.21.15.141:0/2766075208 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df800cd20 con 0x7f9e100d03a0 2024-01-15T11:24:51.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e16475700 1 -- 172.21.15.141:0/2766075208 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e100cdba0 con 0x7f9e100d03a0 2024-01-15T11:24:51.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e0d7fa700 1 -- 172.21.15.141:0/2766075208 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9df8005d20 con 0x7f9e100d03a0 2024-01-15T11:24:51.158 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.153+0000 7f9e0d7fa700 1 -- 172.21.15.141:0/2766075208 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9df80137b0 con 0x7f9e100d03a0 2024-01-15T11:24:51.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.154+0000 7f9e16475700 1 -- 172.21.15.141:0/2766075208 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e10069020 con 0x7f9e100d03a0 2024-01-15T11:24:51.159 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.156+0000 7f9e0d7fa700 1 -- 172.21.15.141:0/2766075208 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f9df8021060 con 0x7f9e100d03a0 2024-01-15T11:24:51.165 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:51.166+0000 7fa553a50200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-15T11:24:51.237 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.240+0000 7f9e16475700 1 -- 172.21.15.141:0/2766075208 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd new", "uuid": "04532dcb-3e36-440a-a9c1-c0d2158ab5e0", "id": 0} v 0) v1 -- 0x7f9e100b6240 con 0x7f9e100d03a0 2024-01-15T11:24:51.240 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.243+0000 7f9e0d7fa700 1 -- 172.21.15.141:0/2766075208 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "04532dcb-3e36-440a-a9c1-c0d2158ab5e0", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9df8017070 con 0x7f9e100d03a0 2024-01-15T11:24:51.241 INFO:teuthology.orchestra.run.smithi141.stdout:0 2024-01-15T11:24:51.242 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.244+0000 7f9e16475700 1 -- 172.21.15.141:0/2766075208 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 msgr2=0x7f9e1014e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.244+0000 7f9e16475700 1 --2- 172.21.15.141:0/2766075208 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 0x7f9e1014e7b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9df800e9b0 tx=0x7f9df800ecc0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.244+0000 7f9e16475700 1 -- 172.21.15.141:0/2766075208 shutdown_connections 2024-01-15T11:24:51.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.244+0000 7f9e16475700 1 --2- 172.21.15.141:0/2766075208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1015f8d0 0x7f9e100b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.244+0000 7f9e16475700 1 --2- 172.21.15.141:0/2766075208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9e100cffc0 0x7f9e101f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.244+0000 7f9e16475700 1 --2- 172.21.15.141:0/2766075208 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9e100d03a0 0x7f9e1014e7b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.244+0000 7f9e16475700 1 -- 172.21.15.141:0/2766075208 >> 172.21.15.141:0/2766075208 conn(0x7f9e100d1e50 msgr2=0x7f9e10070380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:51.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.244+0000 7f9e16475700 1 -- 172.21.15.141:0/2766075208 shutdown_connections 2024-01-15T11:24:51.243 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.244+0000 7f9e16475700 1 -- 172.21.15.141:0/2766075208 wait complete. 2024-01-15T11:24:51.254 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new 17b10b62-1681-487b-87ad-36a5e5e4bb57 1 2024-01-15T11:24:51.276 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:51.278+0000 7f1eb7267200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-15T11:24:51.322 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:51.323+0000 7fa553a50200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-15T11:24:51.396 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.399+0000 7f777c3b6700 1 Processor -- start 2024-01-15T11:24:51.397 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.399+0000 7f777c3b6700 1 -- start start 2024-01-15T11:24:51.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777c3b6700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f777415e110 0x7f777406f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777c3b6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f777406e420 0x7f777406e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.404 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777c3b6700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77740affe0 con 0x7f77740afb30 2024-01-15T11:24:51.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777c3b6700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f777406d980 con 0x7f777415e110 2024-01-15T11:24:51.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777c3b6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77740b0260 con 0x7f777406e420 2024-01-15T11:24:51.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777a152700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f777415e110 0x7f777406f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777a953700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f777406e420 0x7f777406e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777a953700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f777406e420 0x7f777406e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.141:33692/0 (socket says 172.21.15.141:33692) 2024-01-15T11:24:51.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777a152700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f777415e110 0x7f777406f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35860/0 (socket says 172.21.15.141:35860) 2024-01-15T11:24:51.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777a953700 1 -- 172.21.15.141:0/1767603289 learned_addr learned my addr 172.21.15.141:0/1767603289 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-15T11:24:51.405 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777a152700 1 -- 172.21.15.141:0/1767603289 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f777406e420 msgr2=0x7f777406e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:24:51.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777a152700 1 --2- 172.21.15.141:0/1767603289 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f777406e420 0x7f777406e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777a152700 1 -- 172.21.15.141:0/1767603289 >> v1:172.21.15.141:6789/0 conn(0x7f77740afb30 legacy=0x7f777406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f7779150700 1 -- 172.21.15.141:0/1767603289 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443008701 0 0) 0x7f77740affe0 con 0x7f77740afb30 2024-01-15T11:24:51.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777a152700 1 -- 172.21.15.141:0/1767603289 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77740b0540 con 0x7f777415e110 2024-01-15T11:24:51.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.400+0000 7f777a953700 1 --2- 172.21.15.141:0/1767603289 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f777406e420 0x7f777406e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:51.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.401+0000 7f777a152700 1 --2- 172.21.15.141:0/1767603289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f777415e110 0x7f777406f680 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7764009840 tx=0x7f7764009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20e952bd68c946b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:51.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.401+0000 7f7779150700 1 -- 172.21.15.141:0/1767603289 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7764011830 con 0x7f777415e110 2024-01-15T11:24:51.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.401+0000 7f7779150700 1 -- 172.21.15.141:0/1767603289 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7764011990 con 0x7f777415e110 2024-01-15T11:24:51.406 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.401+0000 7f7779150700 1 -- 172.21.15.141:0/1767603289 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7764011d20 con 0x7f777415e110 2024-01-15T11:24:51.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.401+0000 7f777c3b6700 1 -- 172.21.15.141:0/1767603289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f777415e110 msgr2=0x7f777406f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.401+0000 7f777c3b6700 1 --2- 172.21.15.141:0/1767603289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f777415e110 0x7f777406f680 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7764009840 tx=0x7f7764009b50 comp rx=0 tx=0).stop 2024-01-15T11:24:51.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.401+0000 7f777c3b6700 1 -- 172.21.15.141:0/1767603289 shutdown_connections 2024-01-15T11:24:51.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.401+0000 7f777c3b6700 1 --2- 172.21.15.141:0/1767603289 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f777406e420 0x7f777406e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.401+0000 7f777c3b6700 1 --2- 172.21.15.141:0/1767603289 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f777415e110 0x7f777406f680 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.401+0000 7f777c3b6700 1 -- 172.21.15.141:0/1767603289 >> 172.21.15.141:0/1767603289 conn(0x7f77740b3250 msgr2=0x7f77740d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:51.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.401+0000 7f777c3b6700 1 -- 172.21.15.141:0/1767603289 shutdown_connections 2024-01-15T11:24:51.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.401+0000 7f777c3b6700 1 -- 172.21.15.141:0/1767603289 wait complete. 2024-01-15T11:24:51.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.403+0000 7f777c3b6700 1 Processor -- start 2024-01-15T11:24:51.407 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.403+0000 7f777c3b6700 1 -- start start 2024-01-15T11:24:51.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.403+0000 7f777c3b6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f777406e420 0x7f77740cf680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.403+0000 7f777c3b6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77740afb30 0x7f77740c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.403+0000 7f777c3b6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f777415e110 0x7f77741f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.403+0000 7f777c3b6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777415a3c0 con 0x7f777406e420 2024-01-15T11:24:51.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.403+0000 7f777c3b6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77740cfbc0 con 0x7f777415e110 2024-01-15T11:24:51.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.403+0000 7f777c3b6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77741f81c0 con 0x7f77740afb30 2024-01-15T11:24:51.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.403+0000 7f7779951700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77740afb30 0x7f77740c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.408 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.403+0000 7f777a953700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f777415e110 0x7f77741f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f7779951700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77740afb30 0x7f77740c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.141:33698/0 (socket says 172.21.15.141:33698) 2024-01-15T11:24:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f7779951700 1 -- 172.21.15.141:0/2548278807 learned_addr learned my addr 172.21.15.141:0/2548278807 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-15T11:24:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f777a152700 1 --2- 172.21.15.141:0/2548278807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f777406e420 0x7f77740cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f777a953700 1 -- 172.21.15.141:0/2548278807 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77740afb30 msgr2=0x7f77740c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f777a953700 1 --2- 172.21.15.141:0/2548278807 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77740afb30 0x7f77740c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f777a953700 1 -- 172.21.15.141:0/2548278807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f777406e420 msgr2=0x7f77740cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f777a953700 1 --2- 172.21.15.141:0/2548278807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f777406e420 0x7f77740cf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f777a953700 1 -- 172.21.15.141:0/2548278807 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c009b30 con 0x7f777415e110 2024-01-15T11:24:51.409 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f7779951700 1 --2- 172.21.15.141:0/2548278807 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77740afb30 0x7f77740c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:51.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f777a953700 1 --2- 172.21.15.141:0/2548278807 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f777415e110 0x7f77741f72b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f776c00efc0 tx=0x7f776c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:51.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f776affd700 1 -- 172.21.15.141:0/2548278807 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c018070 con 0x7f777415e110 2024-01-15T11:24:51.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f776affd700 1 -- 172.21.15.141:0/2548278807 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c00f040 con 0x7f777415e110 2024-01-15T11:24:51.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f777c3b6700 1 -- 172.21.15.141:0/2548278807 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77640094a0 con 0x7f777415e110 2024-01-15T11:24:51.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f776affd700 1 -- 172.21.15.141:0/2548278807 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c003c60 con 0x7f777415e110 2024-01-15T11:24:51.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f777c3b6700 1 -- 172.21.15.141:0/2548278807 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777406b880 con 0x7f777415e110 2024-01-15T11:24:51.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f776affd700 1 -- 172.21.15.141:0/2548278807 <== mon.1 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f776c00cb80 con 0x7f777415e110 2024-01-15T11:24:51.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.404+0000 7f776affd700 1 -- 172.21.15.141:0/2548278807 <== mon.1 v2:172.21.15.141:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f776c01d7c0 con 0x7f777415e110 2024-01-15T11:24:51.410 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.405+0000 7f777c3b6700 1 -- 172.21.15.141:0/2548278807 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777415a500 con 0x7f777415e110 2024-01-15T11:24:51.411 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.407+0000 7f776affd700 1 -- 172.21.15.141:0/2548278807 <== mon.1 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f776c014ca0 con 0x7f777415e110 2024-01-15T11:24:51.436 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:51.439+0000 7f1eb7267200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-15T11:24:51.472 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:51.473+0000 7fa553a50200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-15T11:24:51.490 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.493+0000 7f777c3b6700 1 -- 172.21.15.141:0/2548278807 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd new", "uuid": "17b10b62-1681-487b-87ad-36a5e5e4bb57", "id": 1} v 0) v1 -- 0x7f7774069020 con 0x7f777415e110 2024-01-15T11:24:51.496 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.499+0000 7f776affd700 1 -- 172.21.15.141:0/2548278807 <== mon.1 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "17b10b62-1681-487b-87ad-36a5e5e4bb57", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f776c02c0b0 con 0x7f777415e110 2024-01-15T11:24:51.496 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-15T11:24:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.500+0000 7f777c3b6700 1 -- 172.21.15.141:0/2548278807 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f777415e110 msgr2=0x7f77741f72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.500+0000 7f777c3b6700 1 --2- 172.21.15.141:0/2548278807 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f777415e110 0x7f77741f72b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f776c00efc0 tx=0x7f776c00c3a0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.500+0000 7f777c3b6700 1 -- 172.21.15.141:0/2548278807 shutdown_connections 2024-01-15T11:24:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.500+0000 7f777c3b6700 1 --2- 172.21.15.141:0/2548278807 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f777415e110 0x7f77741f72b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.500+0000 7f777c3b6700 1 --2- 172.21.15.141:0/2548278807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f777406e420 0x7f77740cf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.500+0000 7f777c3b6700 1 --2- 172.21.15.141:0/2548278807 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77740afb30 0x7f77740c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.500+0000 7f777c3b6700 1 -- 172.21.15.141:0/2548278807 >> 172.21.15.141:0/2548278807 conn(0x7f77740b3250 msgr2=0x7f777415d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.500+0000 7f777c3b6700 1 -- 172.21.15.141:0/2548278807 shutdown_connections 2024-01-15T11:24:51.498 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.500+0000 7f777c3b6700 1 -- 172.21.15.141:0/2548278807 wait complete. 2024-01-15T11:24:51.511 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new 1c190417-61f7-4b0b-888a-cc8bd7c03678 2 2024-01-15T11:24:51.556 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:51.559+0000 7f1eb7267200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-15T11:24:51.579 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:51.581+0000 7fa553a50200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-15T11:24:51.634 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.637+0000 7f3e8721c700 1 Processor -- start 2024-01-15T11:24:51.635 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.637+0000 7f3e8721c700 1 -- start start 2024-01-15T11:24:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e8721c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3e8015bee0 0x7f3e800d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e8721c700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3e800ceb20 0x7f3e8006f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e8721c700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e800d0790 con 0x7f3e800cf020 2024-01-15T11:24:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e8721c700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8006ee70 con 0x7f3e8015bee0 2024-01-15T11:24:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e8721c700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8006f150 con 0x7f3e800ceb20 2024-01-15T11:24:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e857b9700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3e800ceb20 0x7f3e8006f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e857b9700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3e800ceb20 0x7f3e8006f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35884/0 (socket says 172.21.15.141:35884) 2024-01-15T11:24:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e857b9700 1 -- 172.21.15.141:0/1393483000 learned_addr learned my addr 172.21.15.141:0/1393483000 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-15T11:24:51.638 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e84fb8700 1 --2- 172.21.15.141:0/1393483000 >> v2:172.21.15.113:3300/0 conn(0x7f3e8015bee0 0x7f3e800d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e857b9700 1 -- 172.21.15.141:0/1393483000 >> v2:172.21.15.113:3300/0 conn(0x7f3e8015bee0 msgr2=0x7f3e800d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:24:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e857b9700 1 --2- 172.21.15.141:0/1393483000 >> v2:172.21.15.113:3300/0 conn(0x7f3e8015bee0 0x7f3e800d01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e857b9700 1 -- 172.21.15.141:0/1393483000 >> v1:172.21.15.113:6789/0 conn(0x7f3e800cf020 legacy=0x7f3e800ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e857b9700 1 -- 172.21.15.141:0/1393483000 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8006f430 con 0x7f3e800ceb20 2024-01-15T11:24:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.638+0000 7f3e857b9700 1 --2- 172.21.15.141:0/1393483000 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e800ceb20 0x7f3e8006f950 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3e80071e80 tx=0x7f3e7800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=773ed90b213d20bd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.639+0000 7f3e7f7fe700 1 -- 172.21.15.141:0/1393483000 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7800e070 con 0x7f3e800ceb20 2024-01-15T11:24:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.639+0000 7f3e7f7fe700 1 -- 172.21.15.141:0/1393483000 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7800b910 con 0x7f3e800ceb20 2024-01-15T11:24:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.639+0000 7f3e7f7fe700 1 -- 172.21.15.141:0/1393483000 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7800bcc0 con 0x7f3e800ceb20 2024-01-15T11:24:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.639+0000 7f3e8721c700 1 -- 172.21.15.141:0/1393483000 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e800ceb20 msgr2=0x7f3e8006f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.639+0000 7f3e8721c700 1 --2- 172.21.15.141:0/1393483000 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e800ceb20 0x7f3e8006f950 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3e80071e80 tx=0x7f3e7800b330 comp rx=0 tx=0).stop 2024-01-15T11:24:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.639+0000 7f3e8721c700 1 -- 172.21.15.141:0/1393483000 shutdown_connections 2024-01-15T11:24:51.639 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.639+0000 7f3e8721c700 1 --2- 172.21.15.141:0/1393483000 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e800ceb20 0x7f3e8006f950 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.639+0000 7f3e8721c700 1 --2- 172.21.15.141:0/1393483000 >> v2:172.21.15.113:3300/0 conn(0x7f3e8015bee0 0x7f3e800d01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.639+0000 7f3e8721c700 1 -- 172.21.15.141:0/1393483000 >> 172.21.15.141:0/1393483000 conn(0x7f3e800b3780 msgr2=0x7f3e800b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:51.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.639+0000 7f3e8721c700 1 -- 172.21.15.141:0/1393483000 shutdown_connections 2024-01-15T11:24:51.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.639+0000 7f3e8721c700 1 -- 172.21.15.141:0/1393483000 wait complete. 2024-01-15T11:24:51.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.641+0000 7f3e8721c700 1 Processor -- start 2024-01-15T11:24:51.640 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.641+0000 7f3e8721c700 1 -- start start 2024-01-15T11:24:51.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.641+0000 7f3e8721c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3e800ceb20 0x7f3e8015d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.641+0000 7f3e8721c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e800cf020 0x7f3e8006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.641+0000 7f3e8721c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e8015bee0 0x7f3e801e8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.641+0000 7f3e8721c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8015f0b0 con 0x7f3e800cf020 2024-01-15T11:24:51.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.641+0000 7f3e8721c700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8015a670 con 0x7f3e8015bee0 2024-01-15T11:24:51.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.641+0000 7f3e8721c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8006cfb0 con 0x7f3e800ceb20 2024-01-15T11:24:51.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e857b9700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e8015bee0 0x7f3e801e8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e84fb8700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3e800ceb20 0x7f3e8015d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e857b9700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e8015bee0 0x7f3e801e8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35900/0 (socket says 172.21.15.141:35900) 2024-01-15T11:24:51.643 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e857b9700 1 -- 172.21.15.141:0/3505226467 learned_addr learned my addr 172.21.15.141:0/3505226467 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-15T11:24:51.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e7ffff700 1 --2- 172.21.15.141:0/3505226467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e800cf020 0x7f3e8006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e857b9700 1 -- 172.21.15.141:0/3505226467 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3e800ceb20 msgr2=0x7f3e8015d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e857b9700 1 --2- 172.21.15.141:0/3505226467 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3e800ceb20 0x7f3e8015d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e857b9700 1 -- 172.21.15.141:0/3505226467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e800cf020 msgr2=0x7f3e8006ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e857b9700 1 --2- 172.21.15.141:0/3505226467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e800cf020 0x7f3e8006ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e857b9700 1 -- 172.21.15.141:0/3505226467 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e78009b30 con 0x7f3e8015bee0 2024-01-15T11:24:51.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e857b9700 1 --2- 172.21.15.141:0/3505226467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e8015bee0 0x7f3e801e8c80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3e8006f290 tx=0x7f3e78004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:51.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e7d7fa700 1 -- 172.21.15.141:0/3505226467 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e78004880 con 0x7f3e8015bee0 2024-01-15T11:24:51.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e7d7fa700 1 -- 172.21.15.141:0/3505226467 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e780099b0 con 0x7f3e8015bee0 2024-01-15T11:24:51.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e8721c700 1 -- 172.21.15.141:0/3505226467 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8015c880 con 0x7f3e8015bee0 2024-01-15T11:24:51.644 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e7d7fa700 1 -- 172.21.15.141:0/3505226467 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7801b670 con 0x7f3e8015bee0 2024-01-15T11:24:51.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e8721c700 1 -- 172.21.15.141:0/3505226467 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8015d0c0 con 0x7f3e8015bee0 2024-01-15T11:24:51.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e7d7fa700 1 -- 172.21.15.141:0/3505226467 <== mon.1 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3e7801bad0 con 0x7f3e8015bee0 2024-01-15T11:24:51.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.642+0000 7f3e7d7fa700 1 -- 172.21.15.141:0/3505226467 <== mon.1 v2:172.21.15.141:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f3e78024b90 con 0x7f3e8015bee0 2024-01-15T11:24:51.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.643+0000 7f3e8721c700 1 -- 172.21.15.141:0/3505226467 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e800b2b90 con 0x7f3e8015bee0 2024-01-15T11:24:51.645 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.645+0000 7f3e7d7fa700 1 -- 172.21.15.141:0/3505226467 <== mon.1 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f3e78042ce0 con 0x7f3e8015bee0 2024-01-15T11:24:51.657 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:51.660+0000 7f1eb7267200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-15T11:24:51.673 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:51.675+0000 7fa553a50200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-15T11:24:51.726 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.729+0000 7f3e8721c700 1 -- 172.21.15.141:0/3505226467 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1c190417-61f7-4b0b-888a-cc8bd7c03678", "id": 2} v 0) v1 -- 0x7f3e80069020 con 0x7f3e8015bee0 2024-01-15T11:24:51.731 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.734+0000 7f3e7d7fa700 1 -- 172.21.15.141:0/3505226467 <== mon.1 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1c190417-61f7-4b0b-888a-cc8bd7c03678", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3e78031030 con 0x7f3e8015bee0 2024-01-15T11:24:51.731 INFO:teuthology.orchestra.run.smithi141.stdout:2 2024-01-15T11:24:51.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.735+0000 7f3e8721c700 1 -- 172.21.15.141:0/3505226467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e8015bee0 msgr2=0x7f3e801e8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.735+0000 7f3e8721c700 1 --2- 172.21.15.141:0/3505226467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e8015bee0 0x7f3e801e8c80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3e8006f290 tx=0x7f3e78004ad0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.735+0000 7f3e8721c700 1 -- 172.21.15.141:0/3505226467 shutdown_connections 2024-01-15T11:24:51.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.735+0000 7f3e8721c700 1 --2- 172.21.15.141:0/3505226467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e8015bee0 0x7f3e801e8c80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.735+0000 7f3e8721c700 1 --2- 172.21.15.141:0/3505226467 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3e800ceb20 0x7f3e8015d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.735+0000 7f3e8721c700 1 --2- 172.21.15.141:0/3505226467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e800cf020 0x7f3e8006ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.735+0000 7f3e8721c700 1 -- 172.21.15.141:0/3505226467 >> 172.21.15.141:0/3505226467 conn(0x7f3e800b3780 msgr2=0x7f3e800d1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:51.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.735+0000 7f3e8721c700 1 -- 172.21.15.141:0/3505226467 shutdown_connections 2024-01-15T11:24:51.733 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.735+0000 7f3e8721c700 1 -- 172.21.15.141:0/3505226467 wait complete. 2024-01-15T11:24:51.736 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:51.739+0000 7f1eb7267200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-15T11:24:51.744 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new 7f26b85c-4aac-4d47-b9dd-8872d5b5cc11 3 2024-01-15T11:24:51.747 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:51.749+0000 7fa553a50200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-15T11:24:51.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.890+0000 7fda5e214700 1 Processor -- start 2024-01-15T11:24:51.887 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.890+0000 7fda5e214700 1 -- start start 2024-01-15T11:24:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.891+0000 7fda5e214700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fda58166960 0x7fda58073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.891+0000 7fda5e214700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda5806dc40 con 0x7fda580744d0 2024-01-15T11:24:51.889 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.891+0000 7fda5e214700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda580748d0 con 0x7fda58162140 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.891+0000 7fda5e214700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda580b4350 con 0x7fda58166960 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.891+0000 7fda577fe700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fda58162140 0x7fda580b57e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44598/0 (socket says 172.21.15.141:44598) 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.891+0000 7fda577fe700 1 -- 172.21.15.141:0/4249237385 learned_addr learned my addr 172.21.15.141:0/4249237385 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.891+0000 7fda56ffd700 1 --2- 172.21.15.141:0/4249237385 >> v2:172.21.15.113:3300/0 conn(0x7fda58166960 0x7fda58073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.891+0000 7fda567fc700 1 -- 172.21.15.141:0/4249237385 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539719363 0 0) 0x7fda580748d0 con 0x7fda58162140 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.891+0000 7fda567fc700 1 -- 172.21.15.141:0/4249237385 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda400035b0 con 0x7fda58162140 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.892+0000 7fda56ffd700 1 -- 172.21.15.141:0/4249237385 >> v1:172.21.15.141:6789/0 conn(0x7fda58162140 legacy=0x7fda580b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.892+0000 7fda56ffd700 1 -- 172.21.15.141:0/4249237385 >> v1:172.21.15.113:6789/0 conn(0x7fda580744d0 legacy=0x7fda58162520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.892+0000 7fda56ffd700 1 -- 172.21.15.141:0/4249237385 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda580b3d90 con 0x7fda58166960 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.892+0000 7fda56ffd700 1 --2- 172.21.15.141:0/4249237385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda58166960 0x7fda58073f90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009840 tx=0x7fda4c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a648fb576b3d24 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.892+0000 7fda567fc700 1 -- 172.21.15.141:0/4249237385 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c004230 con 0x7fda58166960 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.892+0000 7fda567fc700 1 -- 172.21.15.141:0/4249237385 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c004390 con 0x7fda58166960 2024-01-15T11:24:51.890 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.892+0000 7fda567fc700 1 -- 172.21.15.141:0/4249237385 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c0036c0 con 0x7fda58166960 2024-01-15T11:24:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.893+0000 7fda5e214700 1 -- 172.21.15.141:0/4249237385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda58166960 msgr2=0x7fda58073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.893+0000 7fda5e214700 1 --2- 172.21.15.141:0/4249237385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda58166960 0x7fda58073f90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009840 tx=0x7fda4c009b50 comp rx=0 tx=0).stop 2024-01-15T11:24:51.891 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.893+0000 7fda5e214700 1 -- 172.21.15.141:0/4249237385 shutdown_connections 2024-01-15T11:24:51.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.893+0000 7fda5e214700 1 --2- 172.21.15.141:0/4249237385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda58166960 0x7fda58073f90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.893+0000 7fda5e214700 1 -- 172.21.15.141:0/4249237385 >> 172.21.15.141:0/4249237385 conn(0x7fda580d0c00 msgr2=0x7fda580d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:51.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.893+0000 7fda5e214700 1 -- 172.21.15.141:0/4249237385 shutdown_connections 2024-01-15T11:24:51.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.893+0000 7fda5e214700 1 -- 172.21.15.141:0/4249237385 wait complete. 2024-01-15T11:24:51.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.895+0000 7fda5e214700 1 Processor -- start 2024-01-15T11:24:51.892 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.895+0000 7fda5e214700 1 -- start start 2024-01-15T11:24:51.895 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.896+0000 7fda5e214700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda580744d0 0x7fda58151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.896+0000 7fda5e214700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda58162140 0x7fda5809dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.896+0000 7fda5e214700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fda58166960 0x7fda580b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.896+0000 7fda5e214700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5809ef10 con 0x7fda580744d0 2024-01-15T11:24:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.896+0000 7fda5e214700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5809e670 con 0x7fda58162140 2024-01-15T11:24:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.896+0000 7fda5e214700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5809e950 con 0x7fda58166960 2024-01-15T11:24:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.896+0000 7fda56ffd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda58162140 0x7fda5809dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.896+0000 7fda57fff700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fda58166960 0x7fda580b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.896+0000 7fda56ffd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda58162140 0x7fda5809dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35902/0 (socket says 172.21.15.141:35902) 2024-01-15T11:24:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.896+0000 7fda56ffd700 1 -- 172.21.15.141:0/2858865263 learned_addr learned my addr 172.21.15.141:0/2858865263 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-15T11:24:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.896+0000 7fda577fe700 1 --2- 172.21.15.141:0/2858865263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda580744d0 0x7fda58151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda57fff700 1 -- 172.21.15.141:0/2858865263 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda58162140 msgr2=0x7fda5809dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.896 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda57fff700 1 --2- 172.21.15.141:0/2858865263 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda58162140 0x7fda5809dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda57fff700 1 -- 172.21.15.141:0/2858865263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda580744d0 msgr2=0x7fda58151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda57fff700 1 --2- 172.21.15.141:0/2858865263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda580744d0 0x7fda58151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda57fff700 1 -- 172.21.15.141:0/2858865263 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c0094a0 con 0x7fda58166960 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda577fe700 1 --2- 172.21.15.141:0/2858865263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda580744d0 0x7fda58151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda56ffd700 1 --2- 172.21.15.141:0/2858865263 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda58162140 0x7fda5809dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda57fff700 1 --2- 172.21.15.141:0/2858865263 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fda58166960 0x7fda580b1ad0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fda4800b410 tx=0x7fda4800b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda3ffff700 1 -- 172.21.15.141:0/2858865263 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48011830 con 0x7fda58166960 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda3ffff700 1 -- 172.21.15.141:0/2858865263 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda480119b0 con 0x7fda58166960 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda3ffff700 1 -- 172.21.15.141:0/2858865263 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48011d60 con 0x7fda58166960 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda5e214700 1 -- 172.21.15.141:0/2858865263 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5809ec30 con 0x7fda58166960 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.897+0000 7fda5e214700 1 -- 172.21.15.141:0/2858865263 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda58151d50 con 0x7fda58166960 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.898+0000 7fda3ffff700 1 -- 172.21.15.141:0/2858865263 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fda4801a700 con 0x7fda58166960 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.898+0000 7fda3ffff700 1 -- 172.21.15.141:0/2858865263 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fda48019a50 con 0x7fda58166960 2024-01-15T11:24:51.897 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.898+0000 7fda5e214700 1 -- 172.21.15.141:0/2858865263 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda58162890 con 0x7fda58166960 2024-01-15T11:24:51.899 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.902+0000 7fda3ffff700 1 -- 172.21.15.141:0/2858865263 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fda48039ce0 con 0x7fda58166960 2024-01-15T11:24:51.972 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:51.974+0000 7fa553a50200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-15T11:24:51.974 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:51.977+0000 7f1eb7267200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-15T11:24:51.987 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.990+0000 7fda5e214700 1 -- 172.21.15.141:0/2858865263 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7f26b85c-4aac-4d47-b9dd-8872d5b5cc11", "id": 3} v 0) v1 -- 0x7fda58069020 con 0x7fda58166960 2024-01-15T11:24:51.992 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.995+0000 7fda3ffff700 1 -- 172.21.15.141:0/2858865263 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7f26b85c-4aac-4d47-b9dd-8872d5b5cc11", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fda48028030 con 0x7fda58166960 2024-01-15T11:24:51.992 INFO:teuthology.orchestra.run.smithi141.stdout:3 2024-01-15T11:24:51.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.996+0000 7fda5e214700 1 -- 172.21.15.141:0/2858865263 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fda58166960 msgr2=0x7fda580b1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:51.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.996+0000 7fda5e214700 1 --2- 172.21.15.141:0/2858865263 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fda58166960 0x7fda580b1ad0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fda4800b410 tx=0x7fda4800b720 comp rx=0 tx=0).stop 2024-01-15T11:24:51.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.996+0000 7fda5e214700 1 -- 172.21.15.141:0/2858865263 shutdown_connections 2024-01-15T11:24:51.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.996+0000 7fda5e214700 1 --2- 172.21.15.141:0/2858865263 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fda58166960 0x7fda580b1ad0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.996+0000 7fda5e214700 1 --2- 172.21.15.141:0/2858865263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda580744d0 0x7fda58151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.996+0000 7fda5e214700 1 --2- 172.21.15.141:0/2858865263 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fda58162140 0x7fda5809dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:51.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.996+0000 7fda5e214700 1 -- 172.21.15.141:0/2858865263 >> 172.21.15.141:0/2858865263 conn(0x7fda580d0c00 msgr2=0x7fda580b57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:51.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.996+0000 7fda5e214700 1 -- 172.21.15.141:0/2858865263 shutdown_connections 2024-01-15T11:24:51.994 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:51.996+0000 7fda5e214700 1 -- 172.21.15.141:0/2858865263 wait complete. 2024-01-15T11:24:52.005 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new c44cd675-0510-468f-9e5f-4388ba1c00bc 4 2024-01-15T11:24:52.042 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:52.044+0000 7fa553a50200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-15T11:24:52.051 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:52.054+0000 7f1eb7267200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-15T11:24:52.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.130+0000 7f88fb430700 1 Processor -- start 2024-01-15T11:24:52.128 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.131+0000 7f88fb430700 1 -- start start 2024-01-15T11:24:52.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.131+0000 7f88fb430700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f88f415e110 0x7f88f406f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.131+0000 7f88fb430700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f88f40afb30 0x7f88f406aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.130 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.131+0000 7f88fb430700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f40affe0 con 0x7f88f406b530 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.131+0000 7f88fb430700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f406d980 con 0x7f88f40afb30 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.131+0000 7f88fb430700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f40b0260 con 0x7f88f415e110 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88f91cc700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f88f415e110 0x7f88f406f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88f91cc700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f88f415e110 0x7f88f406f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:35904/0 (socket says 172.21.15.141:35904) 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88f99cd700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f88f406b530 0x7f88f406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:44604/0 (socket says 172.21.15.141:44604) 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88f91cc700 1 -- 172.21.15.141:0/330897171 learned_addr learned my addr 172.21.15.141:0/330897171 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88f89cb700 1 --2- 172.21.15.141:0/330897171 >> v2:172.21.15.113:3300/0 conn(0x7f88f40afb30 0x7f88f406aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88f91cc700 1 -- 172.21.15.141:0/330897171 >> v2:172.21.15.113:3300/0 conn(0x7f88f40afb30 msgr2=0x7f88f406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88f91cc700 1 --2- 172.21.15.141:0/330897171 >> v2:172.21.15.113:3300/0 conn(0x7f88f40afb30 0x7f88f406aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88f91cc700 1 -- 172.21.15.141:0/330897171 >> v1:172.21.15.141:6789/0 conn(0x7f88f406b530 legacy=0x7f88f406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88f91cc700 1 -- 172.21.15.141:0/330897171 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f40b0540 con 0x7f88f415e110 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88ebfff700 1 -- 172.21.15.141:0/330897171 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089519979 0 0) 0x7f88f40affe0 con 0x7f88f406b530 2024-01-15T11:24:52.131 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88f91cc700 1 --2- 172.21.15.141:0/330897171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88f415e110 0x7f88f406f680 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f88f4073030 tx=0x7f88f0009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ed531dc62269d4b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:52.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88ebfff700 1 -- 172.21.15.141:0/330897171 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0011830 con 0x7f88f415e110 2024-01-15T11:24:52.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88ebfff700 1 -- 172.21.15.141:0/330897171 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0011990 con 0x7f88f415e110 2024-01-15T11:24:52.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.132+0000 7f88ebfff700 1 -- 172.21.15.141:0/330897171 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0011d20 con 0x7f88f415e110 2024-01-15T11:24:52.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.133+0000 7f88fb430700 1 -- 172.21.15.141:0/330897171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88f415e110 msgr2=0x7f88f406f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.133+0000 7f88fb430700 1 --2- 172.21.15.141:0/330897171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88f415e110 0x7f88f406f680 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f88f4073030 tx=0x7f88f0009bd0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.133+0000 7f88fb430700 1 -- 172.21.15.141:0/330897171 shutdown_connections 2024-01-15T11:24:52.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.133+0000 7f88fb430700 1 --2- 172.21.15.141:0/330897171 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88f415e110 0x7f88f406f680 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.133+0000 7f88fb430700 1 --2- 172.21.15.141:0/330897171 >> v2:172.21.15.113:3300/0 conn(0x7f88f40afb30 0x7f88f406aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.133+0000 7f88fb430700 1 -- 172.21.15.141:0/330897171 >> 172.21.15.141:0/330897171 conn(0x7f88f40b3250 msgr2=0x7f88f40d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:52.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.133+0000 7f88fb430700 1 -- 172.21.15.141:0/330897171 shutdown_connections 2024-01-15T11:24:52.132 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.133+0000 7f88fb430700 1 -- 172.21.15.141:0/330897171 wait complete. 2024-01-15T11:24:52.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.134+0000 7f88fb430700 1 Processor -- start 2024-01-15T11:24:52.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.134+0000 7f88fb430700 1 -- start start 2024-01-15T11:24:52.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.135+0000 7f88fb430700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f88f406b530 0x7f88f41f7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.135+0000 7f88fb430700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88f40afb30 0x7f88f40cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.135+0000 7f88fb430700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f415e110 0x7f88f40add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.135+0000 7f88fb430700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f41f1050 con 0x7f88f415e110 2024-01-15T11:24:52.135 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.135+0000 7f88fb430700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f41f7b40 con 0x7f88f40afb30 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.135+0000 7f88fb430700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f41f4570 con 0x7f88f406b530 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.135+0000 7f88f91cc700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f88f406b530 0x7f88f41f7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.135+0000 7f88f89cb700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88f40afb30 0x7f88f40cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.135+0000 7f88f91cc700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f88f406b530 0x7f88f41f7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.141:33716/0 (socket says 172.21.15.141:33716) 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.135+0000 7f88f91cc700 1 -- 172.21.15.141:0/4189368627 learned_addr learned my addr 172.21.15.141:0/4189368627 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88f99cd700 1 --2- 172.21.15.141:0/4189368627 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f415e110 0x7f88f40add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88f91cc700 1 -- 172.21.15.141:0/4189368627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88f40afb30 msgr2=0x7f88f40cfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88f91cc700 1 --2- 172.21.15.141:0/4189368627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88f40afb30 0x7f88f40cfdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88f91cc700 1 -- 172.21.15.141:0/4189368627 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f415e110 msgr2=0x7f88f40add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88f91cc700 1 --2- 172.21.15.141:0/4189368627 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f415e110 0x7f88f40add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88f91cc700 1 -- 172.21.15.141:0/4189368627 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f0009550 con 0x7f88f406b530 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88f89cb700 1 --2- 172.21.15.141:0/4189368627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88f40afb30 0x7f88f40cfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:52.136 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88f99cd700 1 --2- 172.21.15.141:0/4189368627 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f415e110 0x7f88f40add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:24:52.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88f91cc700 1 --2- 172.21.15.141:0/4189368627 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f88f406b530 0x7f88f41f7600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009130 tx=0x7f88f0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:52.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88e9ffb700 1 -- 172.21.15.141:0/4189368627 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0004590 con 0x7f88f406b530 2024-01-15T11:24:52.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88e9ffb700 1 -- 172.21.15.141:0/4189368627 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f001e070 con 0x7f88f406b530 2024-01-15T11:24:52.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88fb430700 1 -- 172.21.15.141:0/4189368627 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f41f39f0 con 0x7f88f406b530 2024-01-15T11:24:52.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88e9ffb700 1 -- 172.21.15.141:0/4189368627 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f00038c0 con 0x7f88f406b530 2024-01-15T11:24:52.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.136+0000 7f88fb430700 1 -- 172.21.15.141:0/4189368627 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f41f3fb0 con 0x7f88f406b530 2024-01-15T11:24:52.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.137+0000 7f88fb430700 1 -- 172.21.15.141:0/4189368627 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f40c8e90 con 0x7f88f406b530 2024-01-15T11:24:52.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.137+0000 7f88e9ffb700 1 -- 172.21.15.141:0/4189368627 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f88f0003d20 con 0x7f88f406b530 2024-01-15T11:24:52.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.137+0000 7f88e9ffb700 1 -- 172.21.15.141:0/4189368627 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f88f001a930 con 0x7f88f406b530 2024-01-15T11:24:52.137 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.139+0000 7f88e9ffb700 1 -- 172.21.15.141:0/4189368627 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f88f40c8e90 con 0x7f88f406b530 2024-01-15T11:24:52.221 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.224+0000 7f88fb430700 1 -- 172.21.15.141:0/4189368627 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c44cd675-0510-468f-9e5f-4388ba1c00bc", "id": 4} v 0) v1 -- 0x7f88f41615a0 con 0x7f88f406b530 2024-01-15T11:24:52.226 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.229+0000 7f88e9ffb700 1 -- 172.21.15.141:0/4189368627 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c44cd675-0510-468f-9e5f-4388ba1c00bc", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f88f0020070 con 0x7f88f406b530 2024-01-15T11:24:52.227 INFO:teuthology.orchestra.run.smithi141.stdout:4 2024-01-15T11:24:52.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.230+0000 7f88fb430700 1 -- 172.21.15.141:0/4189368627 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f88f406b530 msgr2=0x7f88f41f7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.230+0000 7f88fb430700 1 --2- 172.21.15.141:0/4189368627 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f88f406b530 0x7f88f41f7600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009130 tx=0x7f88f0004750 comp rx=0 tx=0).stop 2024-01-15T11:24:52.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.230+0000 7f88fb430700 1 -- 172.21.15.141:0/4189368627 shutdown_connections 2024-01-15T11:24:52.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.230+0000 7f88fb430700 1 --2- 172.21.15.141:0/4189368627 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f415e110 0x7f88f40add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.230+0000 7f88fb430700 1 --2- 172.21.15.141:0/4189368627 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f88f406b530 0x7f88f41f7600 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.230+0000 7f88fb430700 1 --2- 172.21.15.141:0/4189368627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f88f40afb30 0x7f88f40cfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.230+0000 7f88fb430700 1 -- 172.21.15.141:0/4189368627 >> 172.21.15.141:0/4189368627 conn(0x7f88f40b3250 msgr2=0x7f88f406ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:52.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.230+0000 7f88fb430700 1 -- 172.21.15.141:0/4189368627 shutdown_connections 2024-01-15T11:24:52.228 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.230+0000 7f88fb430700 1 -- 172.21.15.141:0/4189368627 wait complete. 2024-01-15T11:24:52.239 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new 419abe5e-bcf4-4bd1-8c62-cf481293a40d 5 2024-01-15T11:24:52.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.367+0000 7f1866164700 1 Processor -- start 2024-01-15T11:24:52.364 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.367+0000 7f1866164700 1 -- start start 2024-01-15T11:24:52.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.367+0000 7f1866164700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f18601599b0 0x7f186006edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.367+0000 7f1866164700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f18600b26f0 0x7f186015f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.367+0000 7f1866164700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186015cca0 con 0x7f186006b800 2024-01-15T11:24:52.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.367+0000 7f1866164700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186015c3b0 con 0x7f18601599b0 2024-01-15T11:24:52.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.367+0000 7f1866164700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186015c650 con 0x7f18600b26f0 2024-01-15T11:24:52.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f185ffff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f18600b26f0 0x7f186015f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f1856ffd700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f186006b800 0x7f18600b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42606/0 (socket says 172.21.15.141:42606) 2024-01-15T11:24:52.367 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f185ffff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f18600b26f0 0x7f186015f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:50104/0 (socket says 172.21.15.141:50104) 2024-01-15T11:24:52.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f1856ffd700 1 -- 172.21.15.141:0/3007392470 learned_addr learned my addr 172.21.15.141:0/3007392470 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-15T11:24:52.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f185f7fe700 1 --2- 172.21.15.141:0/3007392470 >> v2:172.21.15.113:3300/0 conn(0x7f18601599b0 0x7f186006edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f185ffff700 1 -- 172.21.15.141:0/3007392470 >> v2:172.21.15.113:3300/0 conn(0x7f18601599b0 msgr2=0x7f186006edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:24:52.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f185ffff700 1 --2- 172.21.15.141:0/3007392470 >> v2:172.21.15.113:3300/0 conn(0x7f18601599b0 0x7f186006edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f185ffff700 1 -- 172.21.15.141:0/3007392470 >> v1:172.21.15.141:6789/0 conn(0x7f186006b800 legacy=0x7f18600b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f185effd700 1 -- 172.21.15.141:0/3007392470 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748881768 0 0) 0x7f186015cca0 con 0x7f186006b800 2024-01-15T11:24:52.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f185ffff700 1 -- 172.21.15.141:0/3007392470 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186015c930 con 0x7f18600b26f0 2024-01-15T11:24:52.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f185ffff700 1 --2- 172.21.15.141:0/3007392470 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18600b26f0 0x7f186015f890 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1860062ac0 tx=0x7f185800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8467b57dd7dc0222 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:52.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f185effd700 1 -- 172.21.15.141:0/3007392470 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185800e070 con 0x7f18600b26f0 2024-01-15T11:24:52.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f185effd700 1 -- 172.21.15.141:0/3007392470 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185800bb20 con 0x7f18600b26f0 2024-01-15T11:24:52.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.368+0000 7f185effd700 1 -- 172.21.15.141:0/3007392470 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858012430 con 0x7f18600b26f0 2024-01-15T11:24:52.368 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.369+0000 7f1866164700 1 -- 172.21.15.141:0/3007392470 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18600b26f0 msgr2=0x7f186015f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.369+0000 7f1866164700 1 --2- 172.21.15.141:0/3007392470 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18600b26f0 0x7f186015f890 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1860062ac0 tx=0x7f185800b330 comp rx=0 tx=0).stop 2024-01-15T11:24:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.369+0000 7f1866164700 1 -- 172.21.15.141:0/3007392470 shutdown_connections 2024-01-15T11:24:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.369+0000 7f1866164700 1 --2- 172.21.15.141:0/3007392470 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f18600b26f0 0x7f186015f890 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.369+0000 7f1866164700 1 --2- 172.21.15.141:0/3007392470 >> v2:172.21.15.113:3300/0 conn(0x7f18601599b0 0x7f186006edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.369+0000 7f1866164700 1 -- 172.21.15.141:0/3007392470 >> 172.21.15.141:0/3007392470 conn(0x7f1860074170 msgr2=0x7f186006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.369+0000 7f1866164700 1 -- 172.21.15.141:0/3007392470 shutdown_connections 2024-01-15T11:24:52.369 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.369+0000 7f1866164700 1 -- 172.21.15.141:0/3007392470 wait complete. 2024-01-15T11:24:52.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.370+0000 7f1866164700 1 Processor -- start 2024-01-15T11:24:52.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.370+0000 7f1866164700 1 -- start start 2024-01-15T11:24:52.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.371+0000 7f1866164700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f186006b800 0x7f1860154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.371+0000 7f1866164700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f18600b26f0 0x7f18601585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.372 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.371+0000 7f1866164700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18601599b0 0x7f18600ab030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.371+0000 7f1866164700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1860155380 con 0x7f18601599b0 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.371+0000 7f1866164700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18600aaa80 con 0x7f186006b800 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.371+0000 7f1866164700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18600a9e70 con 0x7f18600b26f0 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.371+0000 7f185f7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f186006b800 0x7f1860154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.371+0000 7f185f7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f186006b800 0x7f1860154e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:50112/0 (socket says 172.21.15.141:50112) 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.371+0000 7f185f7fe700 1 -- 172.21.15.141:0/1829261446 learned_addr learned my addr 172.21.15.141:0/1829261446 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.371+0000 7f1856ffd700 1 --2- 172.21.15.141:0/1829261446 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f18600b26f0 0x7f18601585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.371+0000 7f185ffff700 1 --2- 172.21.15.141:0/1829261446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18601599b0 0x7f18600ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f1856ffd700 1 -- 172.21.15.141:0/1829261446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f186006b800 msgr2=0x7f1860154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f1856ffd700 1 --2- 172.21.15.141:0/1829261446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f186006b800 0x7f1860154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f1856ffd700 1 -- 172.21.15.141:0/1829261446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18601599b0 msgr2=0x7f18600ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f1856ffd700 1 --2- 172.21.15.141:0/1829261446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18601599b0 0x7f18600ab030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.373 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f1856ffd700 1 -- 172.21.15.141:0/1829261446 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1858009a70 con 0x7f18600b26f0 2024-01-15T11:24:52.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f185f7fe700 1 --2- 172.21.15.141:0/1829261446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f186006b800 0x7f1860154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:52.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f185ffff700 1 --2- 172.21.15.141:0/1829261446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18601599b0 0x7f18600ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:52.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f1856ffd700 1 --2- 172.21.15.141:0/1829261446 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f18600b26f0 0x7f18601585f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f184800c440 tx=0x7f184800c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:52.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f185cff9700 1 -- 172.21.15.141:0/1829261446 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848004c80 con 0x7f18600b26f0 2024-01-15T11:24:52.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f185cff9700 1 -- 172.21.15.141:0/1829261446 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1848004de0 con 0x7f18600b26f0 2024-01-15T11:24:52.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f185cff9700 1 -- 172.21.15.141:0/1829261446 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848007310 con 0x7f18600b26f0 2024-01-15T11:24:52.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f1866164700 1 -- 172.21.15.141:0/1829261446 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18600aa150 con 0x7f18600b26f0 2024-01-15T11:24:52.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f1866164700 1 -- 172.21.15.141:0/1829261446 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18600aa710 con 0x7f18600b26f0 2024-01-15T11:24:52.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f185cff9700 1 -- 172.21.15.141:0/1829261446 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f184801d3f0 con 0x7f18600b26f0 2024-01-15T11:24:52.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.372+0000 7f185cff9700 1 -- 172.21.15.141:0/1829261446 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f184800ce10 con 0x7f18600b26f0 2024-01-15T11:24:52.374 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.373+0000 7f1866164700 1 -- 172.21.15.141:0/1829261446 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1860069020 con 0x7f18600b26f0 2024-01-15T11:24:52.375 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.375+0000 7f185cff9700 1 -- 172.21.15.141:0/1829261446 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f18480134d0 con 0x7f18600b26f0 2024-01-15T11:24:52.414 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:52.415+0000 7fa553a50200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-15T11:24:52.469 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.472+0000 7f1866164700 1 -- 172.21.15.141:0/1829261446 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd new", "uuid": "419abe5e-bcf4-4bd1-8c62-cf481293a40d", "id": 5} v 0) v1 -- 0x7f18600aabc0 con 0x7f18600b26f0 2024-01-15T11:24:52.477 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.480+0000 7f185cff9700 1 -- 172.21.15.141:0/1829261446 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "419abe5e-bcf4-4bd1-8c62-cf481293a40d", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f18480137a0 con 0x7f18600b26f0 2024-01-15T11:24:52.478 INFO:teuthology.orchestra.run.smithi141.stdout:5 2024-01-15T11:24:52.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.481+0000 7f1866164700 1 -- 172.21.15.141:0/1829261446 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f18600b26f0 msgr2=0x7f18601585f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.481+0000 7f1866164700 1 --2- 172.21.15.141:0/1829261446 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f18600b26f0 0x7f18601585f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f184800c440 tx=0x7f184800c750 comp rx=0 tx=0).stop 2024-01-15T11:24:52.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.481+0000 7f1866164700 1 -- 172.21.15.141:0/1829261446 shutdown_connections 2024-01-15T11:24:52.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.481+0000 7f1866164700 1 --2- 172.21.15.141:0/1829261446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18601599b0 0x7f18600ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.481+0000 7f1866164700 1 --2- 172.21.15.141:0/1829261446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f186006b800 0x7f1860154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.481+0000 7f1866164700 1 --2- 172.21.15.141:0/1829261446 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f18600b26f0 0x7f18601585f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.481+0000 7f1866164700 1 -- 172.21.15.141:0/1829261446 >> 172.21.15.141:0/1829261446 conn(0x7f1860074170 msgr2=0x7f186015f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:52.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.481+0000 7f1866164700 1 -- 172.21.15.141:0/1829261446 shutdown_connections 2024-01-15T11:24:52.479 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.481+0000 7f1866164700 1 -- 172.21.15.141:0/1829261446 wait complete. 2024-01-15T11:24:52.481 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:52.484+0000 7f1eb7267200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-15T11:24:52.490 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new 9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0 6 2024-01-15T11:24:52.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.615+0000 7fcac7e8c700 1 Processor -- start 2024-01-15T11:24:52.612 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.615+0000 7fcac7e8c700 1 -- start start 2024-01-15T11:24:52.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac7e8c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac0161690 0x7fcac00d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.614 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac7e8c700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac00d0a70 con 0x7fcac0161a70 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac7e8c700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac00d1c10 con 0x7fcac009b110 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac7e8c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac00b2ab0 con 0x7fcac0161690 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac5c28700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac0161690 0x7fcac00d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac6429700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fcac009b110 0x7fcac00cc4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42614/0 (socket says 172.21.15.141:42614) 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac5c28700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac0161690 0x7fcac00d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.141:32824/0 (socket says 172.21.15.141:32824) 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac6429700 1 -- 172.21.15.141:0/3797010850 learned_addr learned my addr 172.21.15.141:0/3797010850 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac5c28700 1 -- 172.21.15.141:0/3797010850 >> v1:172.21.15.141:6789/0 conn(0x7fcac009b110 legacy=0x7fcac00cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac5c28700 1 -- 172.21.15.141:0/3797010850 >> v1:172.21.15.113:6789/0 conn(0x7fcac0161a70 legacy=0x7fcac009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac4c26700 1 -- 172.21.15.141:0/3797010850 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542702636 0 0) 0x7fcac00d0a70 con 0x7fcac0161a70 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac5c28700 1 -- 172.21.15.141:0/3797010850 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac00b2d90 con 0x7fcac0161690 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac5c28700 1 --2- 172.21.15.141:0/3797010850 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac0161690 0x7fcac00d0f20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcab40097f0 tx=0x7fcab4009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8cdb1a16d5e6c0f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac4c26700 1 -- 172.21.15.141:0/3797010850 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab4004230 con 0x7fcac0161690 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac4c26700 1 -- 172.21.15.141:0/3797010850 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab4004390 con 0x7fcac0161690 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.616+0000 7fcac4c26700 1 -- 172.21.15.141:0/3797010850 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab40036a0 con 0x7fcac0161690 2024-01-15T11:24:52.615 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.617+0000 7fcac7e8c700 1 -- 172.21.15.141:0/3797010850 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac0161690 msgr2=0x7fcac00d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.617+0000 7fcac7e8c700 1 --2- 172.21.15.141:0/3797010850 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac0161690 0x7fcac00d0f20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcab40097f0 tx=0x7fcab4009b00 comp rx=0 tx=0).stop 2024-01-15T11:24:52.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.617+0000 7fcac7e8c700 1 -- 172.21.15.141:0/3797010850 shutdown_connections 2024-01-15T11:24:52.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.617+0000 7fcac7e8c700 1 --2- 172.21.15.141:0/3797010850 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac0161690 0x7fcac00d0f20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.617+0000 7fcac7e8c700 1 -- 172.21.15.141:0/3797010850 >> 172.21.15.141:0/3797010850 conn(0x7fcac00d1e50 msgr2=0x7fcac00d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:52.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.617+0000 7fcac7e8c700 1 -- 172.21.15.141:0/3797010850 shutdown_connections 2024-01-15T11:24:52.616 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.617+0000 7fcac7e8c700 1 -- 172.21.15.141:0/3797010850 wait complete. 2024-01-15T11:24:52.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.618+0000 7fcac7e8c700 1 Processor -- start 2024-01-15T11:24:52.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.618+0000 7fcac7e8c700 1 -- start start 2024-01-15T11:24:52.618 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac7e8c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac009b110 0x7fcac015e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac7e8c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac0161690 0x7fcac00b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac7e8c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcac0161a70 0x7fcac006fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac7e8c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac01f4a60 con 0x7fcac0161a70 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac7e8c700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac00c7c30 con 0x7fcac0161690 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac7e8c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac01f4780 con 0x7fcac009b110 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac5427700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac0161690 0x7fcac00b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac5c28700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac009b110 0x7fcac015e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac5c28700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac009b110 0x7fcac015e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.141:32828/0 (socket says 172.21.15.141:32828) 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac5c28700 1 -- 172.21.15.141:0/1294077569 learned_addr learned my addr 172.21.15.141:0/1294077569 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac6429700 1 --2- 172.21.15.141:0/1294077569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcac0161a70 0x7fcac006fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac5427700 1 -- 172.21.15.141:0/1294077569 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac009b110 msgr2=0x7fcac015e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac5427700 1 --2- 172.21.15.141:0/1294077569 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac009b110 0x7fcac015e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac5427700 1 -- 172.21.15.141:0/1294077569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcac0161a70 msgr2=0x7fcac006fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac5427700 1 --2- 172.21.15.141:0/1294077569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcac0161a70 0x7fcac006fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.619+0000 7fcac5427700 1 -- 172.21.15.141:0/1294077569 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab40094a0 con 0x7fcac0161690 2024-01-15T11:24:52.619 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.620+0000 7fcac5c28700 1 --2- 172.21.15.141:0/1294077569 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac009b110 0x7fcac015e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:52.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.620+0000 7fcac5427700 1 --2- 172.21.15.141:0/1294077569 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac0161690 0x7fcac00b4af0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcac0161e50 tx=0x7fcabc00c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:52.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.620+0000 7fcab27fc700 1 -- 172.21.15.141:0/1294077569 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc0052d0 con 0x7fcac0161690 2024-01-15T11:24:52.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.620+0000 7fcab27fc700 1 -- 172.21.15.141:0/1294077569 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcabc005430 con 0x7fcac0161690 2024-01-15T11:24:52.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.620+0000 7fcac7e8c700 1 -- 172.21.15.141:0/1294077569 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac00c7570 con 0x7fcac0161690 2024-01-15T11:24:52.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.620+0000 7fcab27fc700 1 -- 172.21.15.141:0/1294077569 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc0038a0 con 0x7fcac0161690 2024-01-15T11:24:52.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.620+0000 7fcac7e8c700 1 -- 172.21.15.141:0/1294077569 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac015e980 con 0x7fcac0161690 2024-01-15T11:24:52.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.620+0000 7fcab27fc700 1 -- 172.21.15.141:0/1294077569 <== mon.1 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcabc003d00 con 0x7fcac0161690 2024-01-15T11:24:52.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.620+0000 7fcab27fc700 1 -- 172.21.15.141:0/1294077569 <== mon.1 v2:172.21.15.141:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fcabc0129f0 con 0x7fcac0161690 2024-01-15T11:24:52.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.621+0000 7fcac7e8c700 1 -- 172.21.15.141:0/1294077569 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac00c73d0 con 0x7fcac0161690 2024-01-15T11:24:52.620 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.623+0000 7fcab27fc700 1 -- 172.21.15.141:0/1294077569 <== mon.1 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fcac00c73d0 con 0x7fcac0161690 2024-01-15T11:24:52.631 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:52.633+0000 7fa553a50200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-15T11:24:52.705 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.707+0000 7fcac7e8c700 1 -- 172.21.15.141:0/1294077569 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0", "id": 6} v 0) v1 -- 0x7fcac00c7d70 con 0x7fcac0161690 2024-01-15T11:24:52.713 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.716+0000 7fcab27fc700 1 -- 172.21.15.141:0/1294077569 <== mon.1 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcac00c7d70 con 0x7fcac0161690 2024-01-15T11:24:52.713 INFO:teuthology.orchestra.run.smithi141.stdout:6 2024-01-15T11:24:52.714 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:52.716+0000 7f1eb7267200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-15T11:24:52.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.717+0000 7fcac7e8c700 1 -- 172.21.15.141:0/1294077569 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac0161690 msgr2=0x7fcac00b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.714 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.717+0000 7fcac7e8c700 1 --2- 172.21.15.141:0/1294077569 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac0161690 0x7fcac00b4af0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcac0161e50 tx=0x7fcabc00c6c0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.717+0000 7fcac7e8c700 1 -- 172.21.15.141:0/1294077569 shutdown_connections 2024-01-15T11:24:52.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.717+0000 7fcac7e8c700 1 --2- 172.21.15.141:0/1294077569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcac0161a70 0x7fcac006fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.717+0000 7fcac7e8c700 1 --2- 172.21.15.141:0/1294077569 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcac009b110 0x7fcac015e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.717+0000 7fcac7e8c700 1 --2- 172.21.15.141:0/1294077569 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcac0161690 0x7fcac00b4af0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.717+0000 7fcac7e8c700 1 -- 172.21.15.141:0/1294077569 >> 172.21.15.141:0/1294077569 conn(0x7fcac00d1e50 msgr2=0x7fcac0160330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:52.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.717+0000 7fcac7e8c700 1 -- 172.21.15.141:0/1294077569 shutdown_connections 2024-01-15T11:24:52.715 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.717+0000 7fcac7e8c700 1 -- 172.21.15.141:0/1294077569 wait complete. 2024-01-15T11:24:52.726 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new 5b587b01-e25d-47a8-8bd7-2ec6803cedd7 7 2024-01-15T11:24:52.851 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.854+0000 7ff5d9e7b700 1 Processor -- start 2024-01-15T11:24:52.852 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.854+0000 7ff5d9e7b700 1 -- start start 2024-01-15T11:24:52.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.855+0000 7ff5d9e7b700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff5d415b240 0x7ff5d406e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.855+0000 7ff5d9e7b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff5d406c2e0 0x7ff5d41647f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.855+0000 7ff5d9e7b700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d406eeb0 con 0x7ff5d406e9a0 2024-01-15T11:24:52.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.855+0000 7ff5d9e7b700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5d406f190 con 0x7ff5d406c2e0 2024-01-15T11:24:52.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.855+0000 7ff5d9e7b700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5d406f470 con 0x7ff5d415b240 2024-01-15T11:24:52.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.855+0000 7ff5d37fe700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff5d415b240 0x7ff5d406e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.854 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.855+0000 7ff5d2ffd700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7ff5d406e9a0 0x7ff5d415b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.141:42630/0 (socket says 172.21.15.141:42630) 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.855+0000 7ff5d2ffd700 1 -- 172.21.15.141:0/3150071201 learned_addr learned my addr 172.21.15.141:0/3150071201 (peer_addr_for_me v1:172.21.15.141:0/0) 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.855+0000 7ff5d3fff700 1 --2- 172.21.15.141:0/3150071201 >> v2:172.21.15.113:3300/0 conn(0x7ff5d406c2e0 0x7ff5d41647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d37fe700 1 -- 172.21.15.141:0/3150071201 >> v2:172.21.15.113:3300/0 conn(0x7ff5d406c2e0 msgr2=0x7ff5d41647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d37fe700 1 --2- 172.21.15.141:0/3150071201 >> v2:172.21.15.113:3300/0 conn(0x7ff5d406c2e0 0x7ff5d41647f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d37fe700 1 -- 172.21.15.141:0/3150071201 >> v1:172.21.15.141:6789/0 conn(0x7ff5d406e9a0 legacy=0x7ff5d415b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d27fc700 1 -- 172.21.15.141:0/3150071201 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151984847 0 0) 0x7ff5d406eeb0 con 0x7ff5d406e9a0 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d37fe700 1 -- 172.21.15.141:0/3150071201 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d406f750 con 0x7ff5d415b240 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d37fe700 1 --2- 172.21.15.141:0/3150071201 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d415b240 0x7ff5d406e460 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff5d406db10 tx=0x7ff5c400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f483cb201f209c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d27fc700 1 -- 172.21.15.141:0/3150071201 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c400e070 con 0x7ff5d415b240 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d27fc700 1 -- 172.21.15.141:0/3150071201 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c400bb20 con 0x7ff5d415b240 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d27fc700 1 -- 172.21.15.141:0/3150071201 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4012430 con 0x7ff5d415b240 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/3150071201 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d415b240 msgr2=0x7ff5d406e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d9e7b700 1 --2- 172.21.15.141:0/3150071201 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d415b240 0x7ff5d406e460 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff5d406db10 tx=0x7ff5c400b330 comp rx=0 tx=0).stop 2024-01-15T11:24:52.855 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/3150071201 shutdown_connections 2024-01-15T11:24:52.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d9e7b700 1 --2- 172.21.15.141:0/3150071201 >> v2:172.21.15.113:3300/0 conn(0x7ff5d406c2e0 0x7ff5d41647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d9e7b700 1 --2- 172.21.15.141:0/3150071201 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d415b240 0x7ff5d406e460 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.856+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/3150071201 >> 172.21.15.141:0/3150071201 conn(0x7ff5d40d0c00 msgr2=0x7ff5d40d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:52.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.857+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/3150071201 shutdown_connections 2024-01-15T11:24:52.856 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.857+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/3150071201 wait complete. 2024-01-15T11:24:52.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.858+0000 7ff5d9e7b700 1 Processor -- start 2024-01-15T11:24:52.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.858+0000 7ff5d9e7b700 1 -- start start 2024-01-15T11:24:52.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d9e7b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d406c2e0 0x7ff5d4139300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d9e7b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5d406e9a0 0x7ff5d40b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d9e7b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5d415b240 0x7ff5d40b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:52.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d9e7b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d413a5e0 con 0x7ff5d406e9a0 2024-01-15T11:24:52.858 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d9e7b700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d4153dd0 con 0x7ff5d406c2e0 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d9e7b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d4139c40 con 0x7ff5d415b240 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d37fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d406c2e0 0x7ff5d4139300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d37fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d406c2e0 0x7ff5d4139300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:50138/0 (socket says 172.21.15.141:50138) 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d37fe700 1 -- 172.21.15.141:0/1554902068 learned_addr learned my addr 172.21.15.141:0/1554902068 (peer_addr_for_me v2:172.21.15.141:0/0) 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d2ffd700 1 --2- 172.21.15.141:0/1554902068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5d406e9a0 0x7ff5d40b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d3fff700 1 --2- 172.21.15.141:0/1554902068 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5d415b240 0x7ff5d40b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d2ffd700 1 -- 172.21.15.141:0/1554902068 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5d415b240 msgr2=0x7ff5d40b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d2ffd700 1 --2- 172.21.15.141:0/1554902068 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5d415b240 0x7ff5d40b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d2ffd700 1 -- 172.21.15.141:0/1554902068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d406c2e0 msgr2=0x7ff5d4139300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d2ffd700 1 --2- 172.21.15.141:0/1554902068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d406c2e0 0x7ff5d4139300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d2ffd700 1 -- 172.21.15.141:0/1554902068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c4009a70 con 0x7ff5d406e9a0 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d37fe700 1 --2- 172.21.15.141:0/1554902068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d406c2e0 0x7ff5d4139300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d2ffd700 1 --2- 172.21.15.141:0/1554902068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5d406e9a0 0x7ff5d40b1500 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00c4b0 tx=0x7ff5bc00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5cbfff700 1 -- 172.21.15.141:0/1554902068 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc004c80 con 0x7ff5d406e9a0 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5cbfff700 1 -- 172.21.15.141:0/1554902068 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc004e00 con 0x7ff5d406e9a0 2024-01-15T11:24:52.859 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5cbfff700 1 -- 172.21.15.141:0/1554902068 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc007310 con 0x7ff5d406e9a0 2024-01-15T11:24:52.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/1554902068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d4139f20 con 0x7ff5d406e9a0 2024-01-15T11:24:52.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.859+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/1554902068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d4139840 con 0x7ff5d406e9a0 2024-01-15T11:24:52.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.860+0000 7ff5cbfff700 1 -- 172.21.15.141:0/1554902068 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff5bc01d3f0 con 0x7ff5d406e9a0 2024-01-15T11:24:52.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.860+0000 7ff5cbfff700 1 -- 172.21.15.141:0/1554902068 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7ff5bc01dd00 con 0x7ff5d406e9a0 2024-01-15T11:24:52.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.861+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/1554902068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d413a720 con 0x7ff5d406e9a0 2024-01-15T11:24:52.860 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.863+0000 7ff5cbfff700 1 -- 172.21.15.141:0/1554902068 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff5d413a720 con 0x7ff5d406e9a0 2024-01-15T11:24:52.863 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:52.865+0000 7fa553a50200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-15T11:24:52.941 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:52.943+0000 7fa553a50200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-15T11:24:52.946 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.949+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/1554902068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5b587b01-e25d-47a8-8bd7-2ec6803cedd7", "id": 7} v 0) v1 -- 0x7ff5d4069020 con 0x7ff5d406e9a0 2024-01-15T11:24:52.950 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.953+0000 7ff5cbfff700 1 -- 172.21.15.141:0/1554902068 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5b587b01-e25d-47a8-8bd7-2ec6803cedd7", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff5bc01d820 con 0x7ff5d406e9a0 2024-01-15T11:24:52.950 INFO:teuthology.orchestra.run.smithi141.stdout:7 2024-01-15T11:24:52.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.954+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/1554902068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5d406e9a0 msgr2=0x7ff5d40b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:52.951 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.954+0000 7ff5d9e7b700 1 --2- 172.21.15.141:0/1554902068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5d406e9a0 0x7ff5d40b1500 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00c4b0 tx=0x7ff5bc00c7c0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.954+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/1554902068 shutdown_connections 2024-01-15T11:24:52.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.954+0000 7ff5d9e7b700 1 --2- 172.21.15.141:0/1554902068 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5d415b240 0x7ff5d40b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.954+0000 7ff5d9e7b700 1 --2- 172.21.15.141:0/1554902068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5d406c2e0 0x7ff5d4139300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.954+0000 7ff5d9e7b700 1 --2- 172.21.15.141:0/1554902068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5d406e9a0 0x7ff5d40b1500 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:52.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.954+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/1554902068 >> 172.21.15.141:0/1554902068 conn(0x7ff5d40d0c00 msgr2=0x7ff5d40d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:52.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.954+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/1554902068 shutdown_connections 2024-01-15T11:24:52.952 INFO:teuthology.orchestra.run.smithi141.stderr:2024-01-15T11:24:52.954+0000 7ff5d9e7b700 1 -- 172.21.15.141:0/1554902068 wait complete. 2024-01-15T11:24:52.963 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-15T11:24:52.964 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-15T11:24:52.964 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:52.966+0000 7f1eb7267200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-15T11:24:52.966 INFO:tasks.ceph.osd.0:Started 2024-01-15T11:24:52.966 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-15T11:24:52.966 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-15T11:24:52.968 INFO:tasks.ceph.osd.1:Started 2024-01-15T11:24:52.968 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-15T11:24:52.968 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-15T11:24:52.970 INFO:tasks.ceph.osd.2:Started 2024-01-15T11:24:52.970 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-15T11:24:52.970 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-01-15T11:24:52.972 INFO:tasks.ceph.osd.3:Started 2024-01-15T11:24:52.972 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-15T11:24:52.972 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-01-15T11:24:52.974 INFO:tasks.ceph.osd.4:Started 2024-01-15T11:24:52.975 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-15T11:24:52.975 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-01-15T11:24:52.976 INFO:tasks.ceph.osd.5:Started 2024-01-15T11:24:52.977 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-15T11:24:52.977 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-01-15T11:24:52.978 INFO:tasks.ceph.osd.6:Started 2024-01-15T11:24:52.978 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-15T11:24:52.979 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-01-15T11:24:52.982 INFO:tasks.ceph.osd.7:Started 2024-01-15T11:24:52.983 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:24:53.076 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:53.079+0000 7f1eb7267200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-15T11:24:53.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.161+0000 7f1f730a0700 1 Processor -- start 2024-01-15T11:24:53.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.161+0000 7f1f730a0700 1 -- start start 2024-01-15T11:24:53.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.161+0000 7f1f730a0700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1f6c0ce4d0 0x7f1f6c0d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:53.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.161+0000 7f1f730a0700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6c0d0150 con 0x7f1f6c0ce910 2024-01-15T11:24:53.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.161+0000 7f1f730a0700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6c0cee50 con 0x7f1f6c15e080 2024-01-15T11:24:53.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.161+0000 7f1f730a0700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c15e460 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f7163d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f1f6c0ce910 0x7f1f6c0d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:49242/0 (socket says 172.21.15.113:49242) 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f7163d700 1 -- 172.21.15.113:0/2779090770 learned_addr learned my addr 172.21.15.113:0/2779090770 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f6bfff700 1 --2- 172.21.15.113:0/2779090770 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1f6c0ce4d0 0x7f1f6c0d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f6b7fe700 1 -- 172.21.15.113:0/2779090770 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686016055 0 0) 0x7f1f6c0d0150 con 0x7f1f6c0ce910 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f6b7fe700 1 -- 172.21.15.113:0/2779090770 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f5c0035b0 con 0x7f1f6c0ce910 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f6b7fe700 1 -- 172.21.15.113:0/2779090770 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285165389 0 0) 0x7f1f6c0cee50 con 0x7f1f6c15e080 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f6b7fe700 1 -- 172.21.15.113:0/2779090770 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c0d0150 con 0x7f1f6c15e080 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f6bfff700 1 -- 172.21.15.113:0/2779090770 >> v1:172.21.15.141:6789/0 conn(0x7f1f6c15e080 legacy=0x7f1f6c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f6bfff700 1 -- 172.21.15.113:0/2779090770 >> v1:172.21.15.113:6789/0 conn(0x7f1f6c0ce910 legacy=0x7f1f6c0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f6bfff700 1 -- 172.21.15.113:0/2779090770 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c0cf130 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f6bfff700 1 --2- 172.21.15.113:0/2779090770 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1f6c0ce4d0 0x7f1f6c0d1be0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009870 tx=0x7f1f58009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0be0d4dd7c18508 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f6b7fe700 1 -- 172.21.15.113:0/2779090770 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58011830 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f6b7fe700 1 -- 172.21.15.113:0/2779090770 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f58011990 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.162+0000 7f1f6b7fe700 1 -- 172.21.15.113:0/2779090770 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58011d20 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.163+0000 7f1f730a0700 1 -- 172.21.15.113:0/2779090770 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1f6c0ce4d0 msgr2=0x7f1f6c0d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.163+0000 7f1f730a0700 1 --2- 172.21.15.113:0/2779090770 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1f6c0ce4d0 0x7f1f6c0d1be0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009870 tx=0x7f1f58009b80 comp rx=0 tx=0).stop 2024-01-15T11:24:53.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.163+0000 7f1f730a0700 1 -- 172.21.15.113:0/2779090770 shutdown_connections 2024-01-15T11:24:53.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.163+0000 7f1f730a0700 1 --2- 172.21.15.113:0/2779090770 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1f6c0ce4d0 0x7f1f6c0d1be0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:53.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.163+0000 7f1f730a0700 1 -- 172.21.15.113:0/2779090770 >> 172.21.15.113:0/2779090770 conn(0x7f1f6c073130 msgr2=0x7f1f6c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:53.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.163+0000 7f1f730a0700 1 -- 172.21.15.113:0/2779090770 shutdown_connections 2024-01-15T11:24:53.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.163+0000 7f1f730a0700 1 -- 172.21.15.113:0/2779090770 wait complete. 2024-01-15T11:24:53.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.164+0000 7f1f730a0700 1 Processor -- start 2024-01-15T11:24:53.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.164+0000 7f1f730a0700 1 -- start start 2024-01-15T11:24:53.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f730a0700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1f6c0ce4d0 0x7f1f6c0b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:53.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f730a0700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1f6c0ce910 0x7f1f6c0c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:53.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f730a0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f6c15e080 0x7f1f6c157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:53.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f730a0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c0c9cf0 con 0x7f1f6c15e080 2024-01-15T11:24:53.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f730a0700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c0c9fd0 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f730a0700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c0c5800 con 0x7f1f6c0ce910 2024-01-15T11:24:53.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f7163d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f6c15e080 0x7f1f6c157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:53.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f70e3c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1f6c0ce4d0 0x7f1f6c0b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:53.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f7163d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f6c15e080 0x7f1f6c157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42264/0 (socket says 172.21.15.113:42264) 2024-01-15T11:24:53.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f7163d700 1 -- 172.21.15.113:0/3981337494 learned_addr learned my addr 172.21.15.113:0/3981337494 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f6bfff700 1 --2- 172.21.15.113:0/3981337494 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1f6c0ce910 0x7f1f6c0c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f70e3c700 1 -- 172.21.15.113:0/3981337494 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1f6c0ce910 msgr2=0x7f1f6c0c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f70e3c700 1 --2- 172.21.15.113:0/3981337494 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1f6c0ce910 0x7f1f6c0c8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f70e3c700 1 -- 172.21.15.113:0/3981337494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f6c15e080 msgr2=0x7f1f6c157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f70e3c700 1 --2- 172.21.15.113:0/3981337494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f6c15e080 0x7f1f6c157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f70e3c700 1 -- 172.21.15.113:0/3981337494 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f580094a0 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f7163d700 1 --2- 172.21.15.113:0/3981337494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f6c15e080 0x7f1f6c157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f6bfff700 1 --2- 172.21.15.113:0/3981337494 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1f6c0ce910 0x7f1f6c0c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f70e3c700 1 --2- 172.21.15.113:0/3981337494 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1f6c0ce4d0 0x7f1f6c0b58b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000c3c0 tx=0x7f1f6000c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f697fa700 1 -- 172.21.15.113:0/3981337494 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f600052d0 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f697fa700 1 -- 172.21.15.113:0/3981337494 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f60005430 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f697fa700 1 -- 172.21.15.113:0/3981337494 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f600038c0 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.166+0000 7f1f730a0700 1 -- 172.21.15.113:0/3981337494 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6c0c5ae0 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.167+0000 7f1f730a0700 1 -- 172.21.15.113:0/3981337494 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6c0c85b0 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.167+0000 7f1f697fa700 1 -- 172.21.15.113:0/3981337494 <== mon.1 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1f60003d20 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.167+0000 7f1f697fa700 1 -- 172.21.15.113:0/3981337494 <== mon.1 v2:172.21.15.141:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f1f60012420 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.168+0000 7f1f730a0700 1 -- 172.21.15.113:0/3981337494 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6c0c6000 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.170+0000 7f1f697fa700 1 -- 172.21.15.113:0/3981337494 <== mon.1 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f1f6c0c6000 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.240+0000 7f1f730a0700 1 -- 172.21.15.113:0/3981337494 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f6c067f60 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.240+0000 7f1f697fa700 1 -- 172.21.15.113:0/3981337494 <== mon.1 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f1f60017070 con 0x7f1f6c0ce4d0 2024-01-15T11:24:53.239 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:24:53.239 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":10,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:24:52.950196+0000","last_up_change":"0.000000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:24:53.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.241+0000 7f1f730a0700 1 -- 172.21.15.113:0/3981337494 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1f6c0ce4d0 msgr2=0x7f1f6c0b58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:53.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.241+0000 7f1f730a0700 1 --2- 172.21.15.113:0/3981337494 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1f6c0ce4d0 0x7f1f6c0b58b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000c3c0 tx=0x7f1f6000c6d0 comp rx=0 tx=0).stop 2024-01-15T11:24:53.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.241+0000 7f1f730a0700 1 -- 172.21.15.113:0/3981337494 shutdown_connections 2024-01-15T11:24:53.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.241+0000 7f1f730a0700 1 --2- 172.21.15.113:0/3981337494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f6c15e080 0x7f1f6c157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:53.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.241+0000 7f1f730a0700 1 --2- 172.21.15.113:0/3981337494 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1f6c0ce4d0 0x7f1f6c0b58b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:53.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.241+0000 7f1f730a0700 1 --2- 172.21.15.113:0/3981337494 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1f6c0ce910 0x7f1f6c0c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:53.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.241+0000 7f1f730a0700 1 -- 172.21.15.113:0/3981337494 >> 172.21.15.113:0/3981337494 conn(0x7f1f6c073130 msgr2=0x7f1f6c0cc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:53.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.241+0000 7f1f730a0700 1 -- 172.21.15.113:0/3981337494 shutdown_connections 2024-01-15T11:24:53.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:24:53.241+0000 7f1f730a0700 1 -- 172.21.15.113:0/3981337494 wait complete. 2024-01-15T11:24:53.251 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-15T11:24:53.252 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-15T11:24:53.583 INFO:tasks.ceph.osd.0.smithi113.stderr:2024-01-15T11:24:53.584+0000 7f34bdcf3640 -1 Falling back to public interface 2024-01-15T11:24:53.584 INFO:tasks.ceph.osd.4.smithi141.stderr:2024-01-15T11:24:53.587+0000 7f64cb3b2640 -1 Falling back to public interface 2024-01-15T11:24:53.589 INFO:tasks.ceph.osd.5.smithi141.stderr:2024-01-15T11:24:53.592+0000 7ff021e21640 -1 Falling back to public interface 2024-01-15T11:24:53.590 INFO:tasks.ceph.osd.3.smithi113.stderr:2024-01-15T11:24:53.592+0000 7eff7be2a640 -1 Falling back to public interface 2024-01-15T11:24:53.592 INFO:tasks.ceph.osd.7.smithi141.stderr:2024-01-15T11:24:53.595+0000 7f84a75c7640 -1 Falling back to public interface 2024-01-15T11:24:53.594 INFO:tasks.ceph.osd.2.smithi113.stderr:2024-01-15T11:24:53.596+0000 7f110cadb640 -1 Falling back to public interface 2024-01-15T11:24:53.598 INFO:tasks.ceph.osd.1.smithi113.stderr:2024-01-15T11:24:53.600+0000 7f4139041640 -1 Falling back to public interface 2024-01-15T11:24:53.599 INFO:tasks.ceph.osd.6.smithi141.stderr:2024-01-15T11:24:53.602+0000 7f5981fbb640 -1 Falling back to public interface 2024-01-15T11:24:53.706 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:53.708+0000 7fa553a50200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-15T11:24:53.786 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:53.788+0000 7fa553a50200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-15T11:24:53.789 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:53.792+0000 7f1eb7267200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-15T11:24:53.870 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:53.873+0000 7f1eb7267200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-15T11:24:53.920 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:53.921+0000 7fa553a50200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-15T11:24:54.005 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:54.008+0000 7f1eb7267200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-15T11:24:54.056 DEBUG:teuthology.orchestra.run.smithi113:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-15T11:24:54.173 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.174+0000 7f1412de6700 1 Processor -- start 2024-01-15T11:24:54.173 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.174+0000 7f1412de6700 1 -- start start 2024-01-15T11:24:54.175 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.175+0000 7f1412de6700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f140c0aff30 0x7f140c06c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:54.175 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.175+0000 7f1412de6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f140c0b0390 0x7f140c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:54.175 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.175+0000 7f1412de6700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f140c15ec90 con 0x7f140c158910 2024-01-15T11:24:54.175 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.175+0000 7f1412de6700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140c158cf0 con 0x7f140c0aff30 2024-01-15T11:24:54.175 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.175+0000 7f1412de6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140c15ef70 con 0x7f140c0b0390 2024-01-15T11:24:54.175 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.175+0000 7f1410b82700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f140c158910 0x7f140c0b48e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:49254/0 (socket says 172.21.15.113:49254) 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.175+0000 7f1410b82700 1 -- 172.21.15.113:0/1269666327 learned_addr learned my addr 172.21.15.113:0/1269666327 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.175+0000 7f140bfff700 1 --2- 172.21.15.113:0/1269666327 >> v2:172.21.15.141:3300/0 conn(0x7f140c0aff30 0x7f140c06c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.175+0000 7f1411383700 1 --2- 172.21.15.113:0/1269666327 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f140c0b0390 0x7f140c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.176+0000 7f140b7fe700 1 -- 172.21.15.113:0/1269666327 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148446875 0 0) 0x7f140c15ec90 con 0x7f140c158910 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.176+0000 7f140b7fe700 1 -- 172.21.15.113:0/1269666327 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f80035b0 con 0x7f140c158910 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.176+0000 7f140bfff700 1 -- 172.21.15.113:0/1269666327 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f140c0b0390 msgr2=0x7f140c06c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.176+0000 7f140bfff700 1 --2- 172.21.15.113:0/1269666327 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f140c0b0390 0x7f140c06c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.176+0000 7f140bfff700 1 -- 172.21.15.113:0/1269666327 >> v1:172.21.15.113:6789/0 conn(0x7f140c158910 legacy=0x7f140c0b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.176+0000 7f140bfff700 1 -- 172.21.15.113:0/1269666327 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140c15f250 con 0x7f140c0aff30 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.176+0000 7f1411383700 1 --2- 172.21.15.113:0/1269666327 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f140c0b0390 0x7f140c06c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.176+0000 7f140bfff700 1 --2- 172.21.15.113:0/1269666327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f140c0aff30 0x7f140c06c000 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f13f4009840 tx=0x7f13f4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a2920703b7e6583 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.176+0000 7f140b7fe700 1 -- 172.21.15.113:0/1269666327 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f4011830 con 0x7f140c0aff30 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.176+0000 7f140b7fe700 1 -- 172.21.15.113:0/1269666327 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f4011990 con 0x7f140c0aff30 2024-01-15T11:24:54.176 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.176+0000 7f140b7fe700 1 -- 172.21.15.113:0/1269666327 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f4011d20 con 0x7f140c0aff30 2024-01-15T11:24:54.177 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.177+0000 7f1412de6700 1 -- 172.21.15.113:0/1269666327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f140c0aff30 msgr2=0x7f140c06c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:54.177 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.177+0000 7f1412de6700 1 --2- 172.21.15.113:0/1269666327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f140c0aff30 0x7f140c06c000 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f13f4009840 tx=0x7f13f4009b50 comp rx=0 tx=0).stop 2024-01-15T11:24:54.177 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.177+0000 7f1412de6700 1 -- 172.21.15.113:0/1269666327 shutdown_connections 2024-01-15T11:24:54.177 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.177+0000 7f1412de6700 1 --2- 172.21.15.113:0/1269666327 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f140c0b0390 0x7f140c06c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:54.177 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.177+0000 7f1412de6700 1 --2- 172.21.15.113:0/1269666327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f140c0aff30 0x7f140c06c000 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:54.177 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.177+0000 7f1412de6700 1 -- 172.21.15.113:0/1269666327 >> 172.21.15.113:0/1269666327 conn(0x7f140c071ae0 msgr2=0x7f140c071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:54.177 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.177+0000 7f1412de6700 1 -- 172.21.15.113:0/1269666327 shutdown_connections 2024-01-15T11:24:54.177 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.177+0000 7f1412de6700 1 -- 172.21.15.113:0/1269666327 wait complete. 2024-01-15T11:24:54.179 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.178+0000 7f1412de6700 1 Processor -- start 2024-01-15T11:24:54.180 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.178+0000 7f1412de6700 1 -- start start 2024-01-15T11:24:54.180 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1412de6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f140c0aff30 0x7f140c160b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:54.180 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1412de6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f140c0b0390 0x7f140c1610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:54.180 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1412de6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f140c158910 0x7f140c15b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:24:54.180 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1412de6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1f8400 con 0x7f140c0aff30 2024-01-15T11:24:54.180 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1412de6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1f7780 con 0x7f140c0b0390 2024-01-15T11:24:54.180 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1412de6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1f7a60 con 0x7f140c158910 2024-01-15T11:24:54.180 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1410b82700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f140c0aff30 0x7f140c160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:54.180 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1410b82700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f140c0aff30 0x7f140c160b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42270/0 (socket says 172.21.15.113:42270) 2024-01-15T11:24:54.180 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1410b82700 1 -- 172.21.15.113:0/3348759124 learned_addr learned my addr 172.21.15.113:0/3348759124 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:24:54.180 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1411383700 1 --2- 172.21.15.113:0/3348759124 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f140c158910 0x7f140c15b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:54.180 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f140bfff700 1 --2- 172.21.15.113:0/3348759124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f140c0b0390 0x7f140c1610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1411383700 1 -- 172.21.15.113:0/3348759124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f140c0b0390 msgr2=0x7f140c1610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1411383700 1 --2- 172.21.15.113:0/3348759124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f140c0b0390 0x7f140c1610c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1411383700 1 -- 172.21.15.113:0/3348759124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f140c0aff30 msgr2=0x7f140c160b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1411383700 1 --2- 172.21.15.113:0/3348759124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f140c0aff30 0x7f140c160b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1411383700 1 -- 172.21.15.113:0/3348759124 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13fc0094a0 con 0x7f140c158910 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f1410b82700 1 --2- 172.21.15.113:0/3348759124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f140c0aff30 0x7f140c160b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.179+0000 7f140bfff700 1 --2- 172.21.15.113:0/3348759124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f140c0b0390 0x7f140c1610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.180+0000 7f1411383700 1 --2- 172.21.15.113:0/3348759124 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f140c158910 0x7f140c15b300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00e910 tx=0x7f13fc00ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.180+0000 7f14097fa700 1 -- 172.21.15.113:0/3348759124 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc004020 con 0x7f140c158910 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.180+0000 7f14097fa700 1 -- 172.21.15.113:0/3348759124 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13fc0041a0 con 0x7f140c158910 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.180+0000 7f14097fa700 1 -- 172.21.15.113:0/3348759124 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc004550 con 0x7f140c158910 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.180+0000 7f1412de6700 1 -- 172.21.15.113:0/3348759124 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f40094a0 con 0x7f140c158910 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.180+0000 7f1412de6700 1 -- 172.21.15.113:0/3348759124 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140c1f8020 con 0x7f140c158910 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.180+0000 7f14097fa700 1 -- 172.21.15.113:0/3348759124 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f13fc0039a0 con 0x7f140c158910 2024-01-15T11:24:54.181 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.180+0000 7f14097fa700 1 -- 172.21.15.113:0/3348759124 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f13fc013d10 con 0x7f140c158910 2024-01-15T11:24:54.182 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.181+0000 7f1412de6700 1 -- 172.21.15.113:0/3348759124 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f140c1f81e0 con 0x7f140c158910 2024-01-15T11:24:54.182 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.183+0000 7f14097fa700 1 -- 172.21.15.113:0/3348759124 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f13fc020060 con 0x7f140c158910 2024-01-15T11:24:54.254 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.256+0000 7f1412de6700 1 -- 172.21.15.113:0/3348759124 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f140c068fb0 con 0x7f140c158910 2024-01-15T11:24:54.255 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.257+0000 7f14097fa700 1 -- 172.21.15.113:0/3348759124 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f13fc017070 con 0x7f140c158910 2024-01-15T11:24:54.255 INFO:teuthology.misc.health.smithi113.stdout: 2024-01-15T11:24:54.255 INFO:teuthology.misc.health.smithi113.stdout:{"epoch":10,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:24:52.950196+0000","last_up_change":"0.000000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:24:54.257 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.258+0000 7f1412de6700 1 -- 172.21.15.113:0/3348759124 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f140c158910 msgr2=0x7f140c15b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:24:54.257 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.258+0000 7f1412de6700 1 --2- 172.21.15.113:0/3348759124 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f140c158910 0x7f140c15b300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00e910 tx=0x7f13fc00ec20 comp rx=0 tx=0).stop 2024-01-15T11:24:54.257 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.258+0000 7f1412de6700 1 -- 172.21.15.113:0/3348759124 shutdown_connections 2024-01-15T11:24:54.257 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.258+0000 7f1412de6700 1 --2- 172.21.15.113:0/3348759124 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f140c158910 0x7f140c15b300 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:54.257 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.258+0000 7f1412de6700 1 --2- 172.21.15.113:0/3348759124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f140c0aff30 0x7f140c160b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:54.257 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.258+0000 7f1412de6700 1 --2- 172.21.15.113:0/3348759124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f140c0b0390 0x7f140c1610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:24:54.257 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.258+0000 7f1412de6700 1 -- 172.21.15.113:0/3348759124 >> 172.21.15.113:0/3348759124 conn(0x7f140c071ae0 msgr2=0x7f140c155750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:24:54.257 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.258+0000 7f1412de6700 1 -- 172.21.15.113:0/3348759124 shutdown_connections 2024-01-15T11:24:54.257 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:24:54.258+0000 7f1412de6700 1 -- 172.21.15.113:0/3348759124 wait complete. 2024-01-15T11:24:54.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-15T11:24:54.484 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:54.486+0000 7fa553a50200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-15T11:24:54.562 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:54.564+0000 7f1eb7267200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-15T11:24:54.648 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:54.650+0000 7fa553a50200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-15T11:24:54.734 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:54.736+0000 7f1eb7267200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-15T11:24:54.812 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:54.814+0000 7fa553a50200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-15T11:24:54.899 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:54.901+0000 7f1eb7267200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-15T11:24:55.620 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:55.621+0000 7fa553a50200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-15T11:24:55.721 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:55.723+0000 7f1eb7267200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-15T11:24:55.844 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:55.846+0000 7fa553a50200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-15T11:24:55.917 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:55.919+0000 7fa553a50200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-15T11:24:55.945 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:55.947+0000 7f1eb7267200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-15T11:24:56.019 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:56.021+0000 7f1eb7267200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-15T11:24:56.084 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:56.086+0000 7fa553a50200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-15T11:24:56.198 INFO:tasks.ceph.mgr.x.smithi141.stderr:2024-01-15T11:24:56.200+0000 7f1eb7267200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-15T11:24:57.111 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:57.113+0000 7fa50ee08700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-15T11:24:57.112 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:24:57.113+0000 7fa50ee08700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-15T11:24:57.128 INFO:tasks.ceph.osd.0.smithi113.stderr:2024-01-15T11:24:57.130+0000 7f34bdcf3640 -1 osd.0 0 log_to_monitors true 2024-01-15T11:24:57.144 INFO:tasks.ceph.osd.3.smithi113.stderr:2024-01-15T11:24:57.146+0000 7eff7be2a640 -1 osd.3 0 log_to_monitors true 2024-01-15T11:24:57.146 INFO:tasks.ceph.osd.2.smithi113.stderr:2024-01-15T11:24:57.147+0000 7f110cadb640 -1 osd.2 0 log_to_monitors true 2024-01-15T11:24:57.152 INFO:tasks.ceph.osd.7.smithi141.stderr:2024-01-15T11:24:57.154+0000 7f84a75c7640 -1 osd.7 0 log_to_monitors true 2024-01-15T11:24:57.152 INFO:tasks.ceph.osd.1.smithi113.stderr:2024-01-15T11:24:57.154+0000 7f4139041640 -1 osd.1 0 log_to_monitors true 2024-01-15T11:24:57.162 INFO:tasks.ceph.osd.4.smithi141.stderr:2024-01-15T11:24:57.163+0000 7f64cb3b2640 -1 osd.4 0 log_to_monitors true 2024-01-15T11:24:57.172 INFO:tasks.ceph.osd.6.smithi141.stderr:2024-01-15T11:24:57.174+0000 7f5981fbb640 -1 osd.6 0 log_to_monitors true 2024-01-15T11:24:57.190 INFO:tasks.ceph.osd.5.smithi141.stderr:2024-01-15T11:24:57.191+0000 7ff021e21640 -1 osd.5 0 log_to_monitors true 2024-01-15T11:25:00.490 INFO:tasks.ceph.osd.0.smithi113.stderr:2024-01-15T11:25:00.491+0000 7f34b18a7700 -1 osd.0 0 waiting for initial osdmap 2024-01-15T11:25:00.504 INFO:tasks.ceph.osd.0.smithi113.stderr:2024-01-15T11:25:00.506+0000 7f34a9c87700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T11:25:00.517 INFO:tasks.ceph.osd.3.smithi113.stderr:2024-01-15T11:25:00.519+0000 7eff6f9de700 -1 osd.3 0 waiting for initial osdmap 2024-01-15T11:25:00.526 INFO:tasks.ceph.osd.3.smithi113.stderr:2024-01-15T11:25:00.528+0000 7eff67dbe700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T11:25:00.541 INFO:tasks.ceph.osd.1.smithi113.stderr:2024-01-15T11:25:00.543+0000 7f412cbf5700 -1 osd.1 0 waiting for initial osdmap 2024-01-15T11:25:00.555 INFO:tasks.ceph.osd.2.smithi113.stderr:2024-01-15T11:25:00.557+0000 7f110068f700 -1 osd.2 0 waiting for initial osdmap 2024-01-15T11:25:00.558 INFO:tasks.ceph.osd.1.smithi113.stderr:2024-01-15T11:25:00.560+0000 7f4124fd5700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T11:25:00.567 INFO:tasks.ceph.osd.2.smithi113.stderr:2024-01-15T11:25:00.569+0000 7f10f8a6f700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T11:25:01.072 DEBUG:teuthology.orchestra.run.smithi113:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:01.165 INFO:tasks.ceph.osd.4.smithi141.stderr:2024-01-15T11:25:01.167+0000 7f64bef66700 -1 osd.4 0 waiting for initial osdmap 2024-01-15T11:25:01.180 INFO:tasks.ceph.osd.4.smithi141.stderr:2024-01-15T11:25:01.182+0000 7f64b7346700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T11:25:01.183 INFO:tasks.ceph.osd.5.smithi141.stderr:2024-01-15T11:25:01.185+0000 7ff0159d5700 -1 osd.5 0 waiting for initial osdmap 2024-01-15T11:25:01.194 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.196+0000 7f4e4e7c4700 1 Processor -- start 2024-01-15T11:25:01.194 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.196+0000 7f4e4e7c4700 1 -- start start 2024-01-15T11:25:01.196 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.196+0000 7f4e4e7c4700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4e4815d110 0x7f4e4815fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.196+0000 7f4e4e7c4700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e481600a0 con 0x7f4e480af780 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.196+0000 7f4e4e7c4700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4815d520 con 0x7f4e480cc580 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.196+0000 7f4e4e7c4700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e48160b70 con 0x7f4e4815d110 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.196+0000 7f4e47fff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f4e480af780 0x7f4e480cbe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:47116/0 (socket says 172.21.15.113:47116) 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.196+0000 7f4e47fff700 1 -- 172.21.15.113:0/2424589979 learned_addr learned my addr 172.21.15.113:0/2424589979 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.197+0000 7f4e4cd61700 1 --2- 172.21.15.113:0/2424589979 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4e4815d110 0x7f4e4815fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.197+0000 7f4e46ffd700 1 -- 172.21.15.113:0/2424589979 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455510483 0 0) 0x7f4e481600a0 con 0x7f4e480af780 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.197+0000 7f4e46ffd700 1 -- 172.21.15.113:0/2424589979 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e340035b0 con 0x7f4e480af780 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.197+0000 7f4e46ffd700 1 -- 172.21.15.113:0/2424589979 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223319606 0 0) 0x7f4e4815d520 con 0x7f4e480cc580 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.197+0000 7f4e46ffd700 1 -- 172.21.15.113:0/2424589979 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e481600a0 con 0x7f4e480cc580 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.197+0000 7f4e4cd61700 1 -- 172.21.15.113:0/2424589979 >> v1:172.21.15.141:6789/0 conn(0x7f4e480cc580 legacy=0x7f4e4815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.197+0000 7f4e4cd61700 1 -- 172.21.15.113:0/2424589979 >> v1:172.21.15.113:6789/0 conn(0x7f4e480af780 legacy=0x7f4e480cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.197+0000 7f4e4cd61700 1 -- 172.21.15.113:0/2424589979 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e481602d0 con 0x7f4e4815d110 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.197+0000 7f4e4cd61700 1 --2- 172.21.15.113:0/2424589979 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4e4815d110 0x7f4e4815fb60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009dc0 tx=0x7f4e4000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39525c2e36fb2f0a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:01.197 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.197+0000 7f4e46ffd700 1 -- 172.21.15.113:0/2424589979 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4000e070 con 0x7f4e4815d110 2024-01-15T11:25:01.198 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.197+0000 7f4e46ffd700 1 -- 172.21.15.113:0/2424589979 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4000bb20 con 0x7f4e4815d110 2024-01-15T11:25:01.198 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.197+0000 7f4e46ffd700 1 -- 172.21.15.113:0/2424589979 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e40012430 con 0x7f4e4815d110 2024-01-15T11:25:01.198 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.198+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/2424589979 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4e4815d110 msgr2=0x7f4e4815fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:01.198 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.198+0000 7f4e4e7c4700 1 --2- 172.21.15.113:0/2424589979 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4e4815d110 0x7f4e4815fb60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009dc0 tx=0x7f4e4000b330 comp rx=0 tx=0).stop 2024-01-15T11:25:01.198 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.198+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/2424589979 shutdown_connections 2024-01-15T11:25:01.198 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.198+0000 7f4e4e7c4700 1 --2- 172.21.15.113:0/2424589979 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4e4815d110 0x7f4e4815fb60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:01.198 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.198+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/2424589979 >> 172.21.15.113:0/2424589979 conn(0x7f4e480726e0 msgr2=0x7f4e48072af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:01.198 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.198+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/2424589979 shutdown_connections 2024-01-15T11:25:01.198 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.198+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/2424589979 wait complete. 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.199+0000 7f4e4e7c4700 1 Processor -- start 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.199+0000 7f4e4e7c4700 1 -- start start 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4e7c4700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e480af780 0x7f4e480ad410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4e7c4700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4e480cc580 0x7f4e4806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4e7c4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e4815d110 0x7f4e481f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4e7c4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4815d9d0 con 0x7f4e4815d110 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4e7c4700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4806de90 con 0x7f4e480af780 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4e7c4700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e480c64b0 con 0x7f4e480cc580 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4cd61700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e4815d110 0x7f4e481f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4cd61700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e4815d110 0x7f4e481f3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55618/0 (socket says 172.21.15.113:55618) 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4cd61700 1 -- 172.21.15.113:0/4212257626 learned_addr learned my addr 172.21.15.113:0/4212257626 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e47fff700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e480af780 0x7f4e480ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e477fe700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4e480cc580 0x7f4e4806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:01.201 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4cd61700 1 -- 172.21.15.113:0/4212257626 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4e480cc580 msgr2=0x7f4e4806e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4cd61700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4e480cc580 0x7f4e4806e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4cd61700 1 -- 172.21.15.113:0/4212257626 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e480af780 msgr2=0x7f4e480ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4cd61700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e480af780 0x7f4e480ad410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4cd61700 1 -- 172.21.15.113:0/4212257626 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e40009a70 con 0x7f4e4815d110 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.200+0000 7f4e4cd61700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e4815d110 0x7f4e481f3b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e40014040 tx=0x7f4e40004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.201+0000 7f4e44ff9700 1 -- 172.21.15.113:0/4212257626 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4001c810 con 0x7f4e4815d110 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.201+0000 7f4e44ff9700 1 -- 172.21.15.113:0/4212257626 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e40020070 con 0x7f4e4815d110 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.201+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/4212257626 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e480c6750 con 0x7f4e4815d110 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.201+0000 7f4e44ff9700 1 -- 172.21.15.113:0/4212257626 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4001cbb0 con 0x7f4e4815d110 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.201+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/4212257626 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4815f770 con 0x7f4e4815d110 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.201+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/4212257626 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e480c6c70 con 0x7f4e4815d110 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.202+0000 7f4e44ff9700 1 -- 172.21.15.113:0/4212257626 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f4e4001cd10 con 0x7f4e4815d110 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.202+0000 7f4e44ff9700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f4e2806b2c0 0x7f4e2806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.202+0000 7f4e44ff9700 1 -- 172.21.15.113:0/4212257626 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f4e40022070 con 0x7f4e4815d110 2024-01-15T11:25:01.202 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.202+0000 7f4e47fff700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f4e2806b2c0 0x7f4e2806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:01.203 INFO:tasks.ceph.osd.5.smithi141.stderr:2024-01-15T11:25:01.204+0000 7ff00ddb5700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T11:25:01.203 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.203+0000 7f4e47fff700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f4e2806b2c0 0x7f4e2806d780 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4e38007fd0 tx=0x7f4e38007e00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:01.205 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.207+0000 7f4e44ff9700 1 -- 172.21.15.113:0/4212257626 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e480c6c70 con 0x7f4e4815d110 2024-01-15T11:25:01.210 INFO:tasks.ceph.osd.7.smithi141.stderr:2024-01-15T11:25:01.212+0000 7f849b17b700 -1 osd.7 0 waiting for initial osdmap 2024-01-15T11:25:01.218 INFO:tasks.ceph.osd.6.smithi141.stderr:2024-01-15T11:25:01.220+0000 7f5975b6f700 -1 osd.6 0 waiting for initial osdmap 2024-01-15T11:25:01.222 INFO:tasks.ceph.osd.7.smithi141.stderr:2024-01-15T11:25:01.224+0000 7f849355b700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T11:25:01.230 INFO:tasks.ceph.osd.6.smithi141.stderr:2024-01-15T11:25:01.232+0000 7f596df4f700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T11:25:01.398 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.400+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/4212257626 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e480c6c70 con 0x7f4e4815d110 2024-01-15T11:25:01.399 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.400+0000 7f4e44ff9700 1 -- 172.21.15.113:0/4212257626 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f4e480c6c70 con 0x7f4e4815d110 2024-01-15T11:25:01.399 INFO:teuthology.misc.health.smithi113.stdout: 2024-01-15T11:25:01.399 INFO:teuthology.misc.health.smithi113.stdout:{"epoch":12,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:24:59.107735+0000","last_up_change":"0.000000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:01.401 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.402+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/4212257626 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f4e2806b2c0 msgr2=0x7f4e2806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:01.401 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.402+0000 7f4e4e7c4700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f4e2806b2c0 0x7f4e2806d780 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4e38007fd0 tx=0x7f4e38007e00 comp rx=0 tx=0).stop 2024-01-15T11:25:01.401 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.402+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/4212257626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e4815d110 msgr2=0x7f4e481f3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:01.401 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.402+0000 7f4e4e7c4700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e4815d110 0x7f4e481f3b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e40014040 tx=0x7f4e40004ad0 comp rx=0 tx=0).stop 2024-01-15T11:25:01.401 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.402+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/4212257626 shutdown_connections 2024-01-15T11:25:01.401 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.402+0000 7f4e4e7c4700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f4e2806b2c0 0x7f4e2806d780 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:01.401 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.402+0000 7f4e4e7c4700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e4815d110 0x7f4e481f3b80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:01.401 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.402+0000 7f4e4e7c4700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4e480af780 0x7f4e480ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:01.401 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.402+0000 7f4e4e7c4700 1 --2- 172.21.15.113:0/4212257626 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4e480cc580 0x7f4e4806e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:01.401 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.402+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/4212257626 >> 172.21.15.113:0/4212257626 conn(0x7f4e480726e0 msgr2=0x7f4e4815dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:01.402 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.402+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/4212257626 shutdown_connections 2024-01-15T11:25:01.402 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:01.402+0000 7f4e4e7c4700 1 -- 172.21.15.113:0/4212257626 wait complete. 2024-01-15T11:25:01.413 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-15T11:25:08.219 DEBUG:teuthology.orchestra.run.smithi113:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:08.344 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.345+0000 7fd89d6b7700 1 Processor -- start 2024-01-15T11:25:08.344 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.345+0000 7fd89d6b7700 1 -- start start 2024-01-15T11:25:08.348 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.346+0000 7fd89d6b7700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd898161610 0x7fd8980d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.348 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.346+0000 7fd89d6b7700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd8981619f0 0x7fd89806bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.348 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.346+0000 7fd89d6b7700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8980b9940 con 0x7fd89806aff0 2024-01-15T11:25:08.348 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.346+0000 7fd89d6b7700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8980b9f00 con 0x7fd8981619f0 2024-01-15T11:25:08.349 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.346+0000 7fd89d6b7700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8980b9c20 con 0x7fd898161610 2024-01-15T11:25:08.349 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd8977fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fd89806aff0 0x7fd898161eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:44216/0 (socket says 172.21.15.113:44216) 2024-01-15T11:25:08.349 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd8977fe700 1 -- 172.21.15.113:0/2744604394 learned_addr learned my addr 172.21.15.113:0/2744604394 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:08.349 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd896ffd700 1 --2- 172.21.15.113:0/2744604394 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd898161610 0x7fd8980d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.349 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd8967fc700 1 --2- 172.21.15.113:0/2744604394 >> v2:172.21.15.141:3300/0 conn(0x7fd8981619f0 0x7fd89806bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.349 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969924748 0 0) 0x7fd8980b9940 con 0x7fd89806aff0 2024-01-15T11:25:08.349 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8800035b0 con 0x7fd89806aff0 2024-01-15T11:25:08.349 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 712482800 0 0) 0x7fd8800035b0 con 0x7fd89806aff0 2024-01-15T11:25:08.349 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd898161610 msgr2=0x7fd8980d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:08.349 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 --2- 172.21.15.113:0/2744604394 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd898161610 0x7fd8980d3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.349 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 >> v2:172.21.15.141:3300/0 conn(0x7fd8981619f0 msgr2=0x7fd89806bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:08.349 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 --2- 172.21.15.113:0/2744604394 >> v2:172.21.15.141:3300/0 conn(0x7fd8981619f0 0x7fd89806bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8980d1b80 con 0x7fd89806aff0 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fd8900019f0 con 0x7fd89806aff0 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 >> v1:172.21.15.113:6789/0 conn(0x7fd89806aff0 legacy=0x7fd898161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 --2- 172.21.15.113:0/2744604394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd880003a00 0x7fd880003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 --2- 172.21.15.113:0/2744604394 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd880004850 0x7fd880004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 --2- 172.21.15.113:0/2744604394 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd898161610 0x7fd880005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8980b9f00 con 0x7fd880003a00 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8980b9c20 con 0x7fd880004850 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88000ba10 con 0x7fd898161610 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd896ffd700 1 --2- 172.21.15.113:0/2744604394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd880003a00 0x7fd880003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd8977fe700 1 --2- 172.21.15.113:0/2744604394 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd898161610 0x7fd880005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.347+0000 7fd8967fc700 1 --2- 172.21.15.113:0/2744604394 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd880004850 0x7fd880004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.350 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd896ffd700 1 -- 172.21.15.113:0/2744604394 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd898161610 msgr2=0x7fd880005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:08.351 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd896ffd700 1 --2- 172.21.15.113:0/2744604394 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd898161610 0x7fd880005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.351 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd896ffd700 1 -- 172.21.15.113:0/2744604394 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd880004850 msgr2=0x7fd880004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:08.351 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd896ffd700 1 --2- 172.21.15.113:0/2744604394 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd880004850 0x7fd880004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.351 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd896ffd700 1 -- 172.21.15.113:0/2744604394 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8800035b0 con 0x7fd880003a00 2024-01-15T11:25:08.351 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd896ffd700 1 --2- 172.21.15.113:0/2744604394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd880003a00 0x7fd880003e20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd88c009b60 tx=0x7fd880005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4060a2848f1a106f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:08.351 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c00e6c0 con 0x7fd880003a00 2024-01-15T11:25:08.351 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88c00e820 con 0x7fd880003a00 2024-01-15T11:25:08.351 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd895ffb700 1 -- 172.21.15.113:0/2744604394 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c00eb90 con 0x7fd880003a00 2024-01-15T11:25:08.351 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd89d6b7700 1 -- 172.21.15.113:0/2744604394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd880003a00 msgr2=0x7fd880003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:08.351 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd89d6b7700 1 --2- 172.21.15.113:0/2744604394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd880003a00 0x7fd880003e20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd88c009b60 tx=0x7fd880005880 comp rx=0 tx=0).stop 2024-01-15T11:25:08.351 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd8977fe700 1 -- 172.21.15.113:0/2744604394 reap_dead start 2024-01-15T11:25:08.351 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd89d6b7700 1 -- 172.21.15.113:0/2744604394 shutdown_connections 2024-01-15T11:25:08.352 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.348+0000 7fd89d6b7700 1 -- 172.21.15.113:0/2744604394 >> 172.21.15.113:0/2744604394 conn(0x7fd8980d44d0 msgr2=0x7fd8980d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:08.352 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.349+0000 7fd89d6b7700 1 -- 172.21.15.113:0/2744604394 shutdown_connections 2024-01-15T11:25:08.352 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.349+0000 7fd89d6b7700 1 -- 172.21.15.113:0/2744604394 wait complete. 2024-01-15T11:25:08.355 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.350+0000 7fd89d6b7700 1 Processor -- start 2024-01-15T11:25:08.355 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.350+0000 7fd89d6b7700 1 -- start start 2024-01-15T11:25:08.355 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.351+0000 7fd89d6b7700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd880003a00 0x7fd8981f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.355 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.351+0000 7fd89d6b7700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8980cbf30 0x7fd8980cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.355 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.351+0000 7fd89d6b7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8981597e0 0x7fd898159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.355 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.351+0000 7fd89d6b7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8980cd6b0 con 0x7fd8981597e0 2024-01-15T11:25:08.355 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.351+0000 7fd89d6b7700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89815a1a0 con 0x7fd8980cbf30 2024-01-15T11:25:08.355 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.351+0000 7fd89d6b7700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89814d0e0 con 0x7fd880003a00 2024-01-15T11:25:08.355 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.351+0000 7fd8977fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8981597e0 0x7fd898159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.355 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.351+0000 7fd8977fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8981597e0 0x7fd898159c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36206/0 (socket says 172.21.15.113:36206) 2024-01-15T11:25:08.355 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.351+0000 7fd8977fe700 1 -- 172.21.15.113:0/246134647 learned_addr learned my addr 172.21.15.113:0/246134647 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.351+0000 7fd8967fc700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8980cbf30 0x7fd8980cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.351+0000 7fd896ffd700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd880003a00 0x7fd8981f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.352+0000 7fd8977fe700 1 -- 172.21.15.113:0/246134647 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd880003a00 msgr2=0x7fd8981f37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.352+0000 7fd8977fe700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd880003a00 0x7fd8981f37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.352+0000 7fd8977fe700 1 -- 172.21.15.113:0/246134647 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8980cbf30 msgr2=0x7fd8980cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.352+0000 7fd8977fe700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8980cbf30 0x7fd8980cc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.352+0000 7fd8977fe700 1 -- 172.21.15.113:0/246134647 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8980ccaa0 con 0x7fd8981597e0 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.352+0000 7fd8977fe700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8981597e0 0x7fd898159c60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd8980d4e70 tx=0x7fd89000a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.352+0000 7fd87f7fe700 1 -- 172.21.15.113:0/246134647 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89000adb0 con 0x7fd8981597e0 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.352+0000 7fd87f7fe700 1 -- 172.21.15.113:0/246134647 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8900133f0 con 0x7fd8981597e0 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.352+0000 7fd89d6b7700 1 -- 172.21.15.113:0/246134647 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8980ccd80 con 0x7fd8981597e0 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.352+0000 7fd87f7fe700 1 -- 172.21.15.113:0/246134647 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8900137a0 con 0x7fd8981597e0 2024-01-15T11:25:08.356 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.352+0000 7fd89d6b7700 1 -- 172.21.15.113:0/246134647 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8980cd340 con 0x7fd8981597e0 2024-01-15T11:25:08.357 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.353+0000 7fd89d6b7700 1 -- 172.21.15.113:0/246134647 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd898068fb0 con 0x7fd8981597e0 2024-01-15T11:25:08.357 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.353+0000 7fd87f7fe700 1 -- 172.21.15.113:0/246134647 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd890013900 con 0x7fd8981597e0 2024-01-15T11:25:08.357 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.354+0000 7fd87f7fe700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd88406b350 0x7fd88406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.357 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.354+0000 7fd896ffd700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd88406b350 0x7fd88406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.357 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.354+0000 7fd87f7fe700 1 -- 172.21.15.113:0/246134647 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd89008a0f0 con 0x7fd8981597e0 2024-01-15T11:25:08.357 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.354+0000 7fd896ffd700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd88406b350 0x7fd88406d810 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd88c0090b0 tx=0x7fd88c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:08.357 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.359+0000 7fd87f7fe700 1 -- 172.21.15.113:0/246134647 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8900552b0 con 0x7fd8981597e0 2024-01-15T11:25:08.549 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.550+0000 7fd89d6b7700 1 -- 172.21.15.113:0/246134647 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8980d1620 con 0x7fd8981597e0 2024-01-15T11:25:08.550 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.551+0000 7fd87f7fe700 1 -- 172.21.15.113:0/246134647 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12910 (secure 0 0 0) 0x7fd8900588d0 con 0x7fd8981597e0 2024-01-15T11:25:08.550 INFO:teuthology.misc.health.smithi113.stdout: 2024-01-15T11:25:08.550 INFO:teuthology.misc.health.smithi113.stdout:{"epoch":16,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:04.497258+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:08.552 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.553+0000 7fd89d6b7700 1 -- 172.21.15.113:0/246134647 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd88406b350 msgr2=0x7fd88406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:08.552 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.553+0000 7fd89d6b7700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd88406b350 0x7fd88406d810 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd88c0090b0 tx=0x7fd88c008030 comp rx=0 tx=0).stop 2024-01-15T11:25:08.552 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.553+0000 7fd89d6b7700 1 -- 172.21.15.113:0/246134647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8981597e0 msgr2=0x7fd898159c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:08.552 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.553+0000 7fd89d6b7700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8981597e0 0x7fd898159c60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd8980d4e70 tx=0x7fd89000a640 comp rx=0 tx=0).stop 2024-01-15T11:25:08.552 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.553+0000 7fd89d6b7700 1 -- 172.21.15.113:0/246134647 shutdown_connections 2024-01-15T11:25:08.552 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.553+0000 7fd89d6b7700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd88406b350 0x7fd88406d810 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.552 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.553+0000 7fd89d6b7700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8981597e0 0x7fd898159c60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.553 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.553+0000 7fd89d6b7700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd880003a00 0x7fd8981f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.553 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.553+0000 7fd89d6b7700 1 --2- 172.21.15.113:0/246134647 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd8980cbf30 0x7fd8980cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.553 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.553+0000 7fd89d6b7700 1 -- 172.21.15.113:0/246134647 >> 172.21.15.113:0/246134647 conn(0x7fd8980d44d0 msgr2=0x7fd898160570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:08.553 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.553+0000 7fd89d6b7700 1 -- 172.21.15.113:0/246134647 shutdown_connections 2024-01-15T11:25:08.553 INFO:teuthology.misc.health.smithi113.stderr:2024-01-15T11:25:08.553+0000 7fd89d6b7700 1 -- 172.21.15.113:0/246134647 wait complete. 2024-01-15T11:25:08.563 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-15T11:25:08.563 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-15T11:25:08.564 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T11:25:08.564 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T11:25:08.566 INFO:tasks.ceph.mds.a:Started 2024-01-15T11:25:08.566 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T11:25:08.566 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T11:25:08.568 INFO:tasks.ceph.mds.c:Started 2024-01-15T11:25:08.568 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T11:25:08.569 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T11:25:08.571 INFO:tasks.ceph.mds.b:Started 2024-01-15T11:25:08.571 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T11:25:08.572 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T11:25:08.574 INFO:tasks.ceph.mds.d:Started 2024-01-15T11:25:08.574 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-01-15T11:25:08.574 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:08.646 INFO:tasks.ceph.mds.a.smithi113.stdout:starting mds.a at 2024-01-15T11:25:08.647 INFO:tasks.ceph.mds.c.smithi113.stdout:starting mds.c at 2024-01-15T11:25:08.663 INFO:tasks.ceph.mds.d.smithi141.stdout:starting mds.d at 2024-01-15T11:25:08.663 INFO:tasks.ceph.mds.b.smithi141.stdout:starting mds.b at 2024-01-15T11:25:08.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.704+0000 7f2bdb1f6700 1 Processor -- start 2024-01-15T11:25:08.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.704+0000 7f2bdb1f6700 1 -- start start 2024-01-15T11:25:08.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.705+0000 7f2bdb1f6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2bd4158670 0x7f2bd40622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.705+0000 7f2bdb1f6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bd4158a50 0x7f2bd40c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.705+0000 7f2bdb1f6700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2bd4162a30 0x7f2bd40c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.705+0000 7f2bdb1f6700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd40cfff0 con 0x7f2bd4158670 2024-01-15T11:25:08.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.705+0000 7f2bdb1f6700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd40cfd10 con 0x7f2bd4162a30 2024-01-15T11:25:08.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.705+0000 7f2bdb1f6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd4161f50 con 0x7f2bd4158a50 2024-01-15T11:25:08.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd8f92700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2bd4158670 0x7f2bd40622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd8f92700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2bd4158670 0x7f2bd40622a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36248/0 (socket says 172.21.15.113:36248) 2024-01-15T11:25:08.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd8f92700 1 -- 172.21.15.113:0/942822836 learned_addr learned my addr 172.21.15.113:0/942822836 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:08.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd9793700 1 --2- 172.21.15.113:0/942822836 >> v2:172.21.15.141:3300/0 conn(0x7f2bd4162a30 0x7f2bd40c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd3fff700 1 --2- 172.21.15.113:0/942822836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bd4158a50 0x7f2bd40c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd8f92700 1 -- 172.21.15.113:0/942822836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bd4158a50 msgr2=0x7f2bd40c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd8f92700 1 --2- 172.21.15.113:0/942822836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bd4158a50 0x7f2bd40c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd8f92700 1 -- 172.21.15.113:0/942822836 >> v2:172.21.15.141:3300/0 conn(0x7f2bd4162a30 msgr2=0x7f2bd40c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd8f92700 1 --2- 172.21.15.113:0/942822836 >> v2:172.21.15.141:3300/0 conn(0x7f2bd4162a30 0x7f2bd40c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd8f92700 1 -- 172.21.15.113:0/942822836 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd4162230 con 0x7f2bd4158670 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd3fff700 1 --2- 172.21.15.113:0/942822836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bd4158a50 0x7f2bd40c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd9793700 1 --2- 172.21.15.113:0/942822836 >> v2:172.21.15.141:3300/0 conn(0x7f2bd4162a30 0x7f2bd40c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd8f92700 1 --2- 172.21.15.113:0/942822836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bd4158670 0x7f2bd40622a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2bd40b0910 tx=0x7f2bc8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c36900bf37e22d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd37fe700 1 -- 172.21.15.113:0/942822836 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc800b870 con 0x7f2bd4158670 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd37fe700 1 -- 172.21.15.113:0/942822836 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc800b9d0 con 0x7f2bd4158670 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.706+0000 7f2bd37fe700 1 -- 172.21.15.113:0/942822836 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc800bd60 con 0x7f2bd4158670 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.707+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/942822836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bd4158670 msgr2=0x7f2bd40622a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.707+0000 7f2bdb1f6700 1 --2- 172.21.15.113:0/942822836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bd4158670 0x7f2bd40622a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2bd40b0910 tx=0x7f2bc8009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.707+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/942822836 shutdown_connections 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.707+0000 7f2bdb1f6700 1 --2- 172.21.15.113:0/942822836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bd4158670 0x7f2bd40622a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.707+0000 7f2bdb1f6700 1 --2- 172.21.15.113:0/942822836 >> v2:172.21.15.141:3300/0 conn(0x7f2bd4162a30 0x7f2bd40c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.707+0000 7f2bdb1f6700 1 --2- 172.21.15.113:0/942822836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bd4158a50 0x7f2bd40c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.707+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/942822836 >> 172.21.15.113:0/942822836 conn(0x7f2bd40b57b0 msgr2=0x7f2bd40b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:08.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.707+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/942822836 shutdown_connections 2024-01-15T11:25:08.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.707+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/942822836 wait complete. 2024-01-15T11:25:08.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.708+0000 7f2bdb1f6700 1 Processor -- start 2024-01-15T11:25:08.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.708+0000 7f2bdb1f6700 1 -- start start 2024-01-15T11:25:08.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bdb1f6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bd4158670 0x7f2bd409a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bdb1f6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bd4158a50 0x7f2bd4146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bdb1f6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bd4162a30 0x7f2bd4149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bdb1f6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd40aec90 con 0x7f2bd4158670 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bdb1f6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd40aded0 con 0x7f2bd4162a30 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bdb1f6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd40ae1b0 con 0x7f2bd4158a50 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bd8f92700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bd4158670 0x7f2bd409a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bd8f92700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bd4158670 0x7f2bd409a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36262/0 (socket says 172.21.15.113:36262) 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bd8f92700 1 -- 172.21.15.113:0/408323537 learned_addr learned my addr 172.21.15.113:0/408323537 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bd3fff700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bd4158a50 0x7f2bd4146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bd9793700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bd4162a30 0x7f2bd4149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bd8f92700 1 -- 172.21.15.113:0/408323537 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bd4158a50 msgr2=0x7f2bd4146dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bd8f92700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bd4158a50 0x7f2bd4146dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bd8f92700 1 -- 172.21.15.113:0/408323537 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bd4162a30 msgr2=0x7f2bd4149820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bd8f92700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bd4162a30 0x7f2bd4149820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bd8f92700 1 -- 172.21.15.113:0/408323537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc8009550 con 0x7f2bd4158670 2024-01-15T11:25:08.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.709+0000 7f2bd8f92700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bd4158670 0x7f2bd409a380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8000c00 tx=0x7f2bc8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:08.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.710+0000 7f2bd17fa700 1 -- 172.21.15.113:0/408323537 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8004540 con 0x7f2bd4158670 2024-01-15T11:25:08.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.710+0000 7f2bd17fa700 1 -- 172.21.15.113:0/408323537 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc80036a0 con 0x7f2bd4158670 2024-01-15T11:25:08.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.710+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/408323537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd40ae490 con 0x7f2bd4158670 2024-01-15T11:25:08.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.710+0000 7f2bd17fa700 1 -- 172.21.15.113:0/408323537 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8003a50 con 0x7f2bd4158670 2024-01-15T11:25:08.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.710+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/408323537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd40adb60 con 0x7f2bd4158670 2024-01-15T11:25:08.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.710+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/408323537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd4067f60 con 0x7f2bd4158670 2024-01-15T11:25:08.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.711+0000 7f2bd17fa700 1 -- 172.21.15.113:0/408323537 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2bc8021070 con 0x7f2bd4158670 2024-01-15T11:25:08.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.711+0000 7f2bd17fa700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2bb806b350 0x7f2bb806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:08.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.711+0000 7f2bd17fa700 1 -- 172.21.15.113:0/408323537 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2bc808aea0 con 0x7f2bd4158670 2024-01-15T11:25:08.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.711+0000 7f2bd3fff700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2bb806b350 0x7f2bb806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:08.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.712+0000 7f2bd3fff700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2bb806b350 0x7f2bb806d810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2bd414df20 tx=0x7f2bc0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:08.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.716+0000 7f2bd17fa700 1 -- 172.21.15.113:0/408323537 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2bc80560f0 con 0x7f2bd4158670 2024-01-15T11:25:08.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.907+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/408323537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bd40cde80 con 0x7f2bd4158670 2024-01-15T11:25:08.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.908+0000 7f2bd17fa700 1 -- 172.21.15.113:0/408323537 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12910 (secure 0 0 0) 0x7f2bc8059710 con 0x7f2bd4158670 2024-01-15T11:25:08.907 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:08.907 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":16,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:04.497258+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:08.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.911+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/408323537 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2bb806b350 msgr2=0x7f2bb806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:08.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.911+0000 7f2bdb1f6700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2bb806b350 0x7f2bb806d810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2bd414df20 tx=0x7f2bc0008040 comp rx=0 tx=0).stop 2024-01-15T11:25:08.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.911+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/408323537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bd4158670 msgr2=0x7f2bd409a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:08.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.911+0000 7f2bdb1f6700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bd4158670 0x7f2bd409a380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8000c00 tx=0x7f2bc8004750 comp rx=0 tx=0).stop 2024-01-15T11:25:08.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.911+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/408323537 shutdown_connections 2024-01-15T11:25:08.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.911+0000 7f2bdb1f6700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2bb806b350 0x7f2bb806d810 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.911+0000 7f2bdb1f6700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bd4162a30 0x7f2bd4149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.911+0000 7f2bdb1f6700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bd4158670 0x7f2bd409a380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.911+0000 7f2bdb1f6700 1 --2- 172.21.15.113:0/408323537 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bd4158a50 0x7f2bd4146dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:08.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.911+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/408323537 >> 172.21.15.113:0/408323537 conn(0x7f2bd40b57b0 msgr2=0x7f2bd40b12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:08.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.911+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/408323537 shutdown_connections 2024-01-15T11:25:08.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:08.911+0000 7f2bdb1f6700 1 -- 172.21.15.113:0/408323537 wait complete. 2024-01-15T11:25:08.921 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:08.922 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:09.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.047+0000 7ff92d6f1700 1 Processor -- start 2024-01-15T11:25:09.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.047+0000 7ff92d6f1700 1 -- start start 2024-01-15T11:25:09.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff92d6f1700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9280b1740 0x7ff92806b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:09.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff92d6f1700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92806bde0 con 0x7ff92815bb30 2024-01-15T11:25:09.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff92d6f1700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff928161d10 con 0x7ff92806b9e0 2024-01-15T11:25:09.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff92d6f1700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff928162990 con 0x7ff9280b1740 2024-01-15T11:25:09.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff926ffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ff92815bb30 0x7ff92815f040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:44246/0 (socket says 172.21.15.113:44246) 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff926ffd700 1 -- 172.21.15.113:0/3676801937 learned_addr learned my addr 172.21.15.113:0/3676801937 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff9267fc700 1 --2- 172.21.15.113:0/3676801937 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9280b1740 0x7ff92806b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff925ffb700 1 -- 172.21.15.113:0/3676801937 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695382840 0 0) 0x7ff92806bde0 con 0x7ff92815bb30 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff925ffb700 1 -- 172.21.15.113:0/3676801937 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9100035b0 con 0x7ff92815bb30 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff925ffb700 1 -- 172.21.15.113:0/3676801937 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152597779 0 0) 0x7ff928161d10 con 0x7ff92806b9e0 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff925ffb700 1 -- 172.21.15.113:0/3676801937 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff92806bde0 con 0x7ff92806b9e0 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff925ffb700 1 -- 172.21.15.113:0/3676801937 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1186662127 0 0) 0x7ff9100035b0 con 0x7ff92815bb30 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff9267fc700 1 -- 172.21.15.113:0/3676801937 >> v1:172.21.15.141:6789/0 conn(0x7ff92806b9e0 legacy=0x7ff9280ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff9267fc700 1 -- 172.21.15.113:0/3676801937 >> v1:172.21.15.113:6789/0 conn(0x7ff92815bb30 legacy=0x7ff92815f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff9267fc700 1 -- 172.21.15.113:0/3676801937 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff928161ff0 con 0x7ff9280b1740 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.048+0000 7ff9267fc700 1 --2- 172.21.15.113:0/3676801937 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9280b1740 0x7ff92806b4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009870 tx=0x7ff91c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b42ee8681681395 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.049+0000 7ff925ffb700 1 -- 172.21.15.113:0/3676801937 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918001670 con 0x7ff9280b1740 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.049+0000 7ff925ffb700 1 -- 172.21.15.113:0/3676801937 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c004030 con 0x7ff9280b1740 2024-01-15T11:25:09.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.049+0000 7ff925ffb700 1 -- 172.21.15.113:0/3676801937 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c0043c0 con 0x7ff9280b1740 2024-01-15T11:25:09.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.049+0000 7ff92d6f1700 1 -- 172.21.15.113:0/3676801937 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9280b1740 msgr2=0x7ff92806b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:09.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.049+0000 7ff92d6f1700 1 --2- 172.21.15.113:0/3676801937 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9280b1740 0x7ff92806b4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009870 tx=0x7ff91c009b80 comp rx=0 tx=0).stop 2024-01-15T11:25:09.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.049+0000 7ff92d6f1700 1 -- 172.21.15.113:0/3676801937 shutdown_connections 2024-01-15T11:25:09.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.049+0000 7ff92d6f1700 1 --2- 172.21.15.113:0/3676801937 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9280b1740 0x7ff92806b4a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.049+0000 7ff92d6f1700 1 -- 172.21.15.113:0/3676801937 >> 172.21.15.113:0/3676801937 conn(0x7ff92809aa00 msgr2=0x7ff92809ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:09.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.049+0000 7ff92d6f1700 1 -- 172.21.15.113:0/3676801937 shutdown_connections 2024-01-15T11:25:09.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.049+0000 7ff92d6f1700 1 -- 172.21.15.113:0/3676801937 wait complete. 2024-01-15T11:25:09.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.050+0000 7ff92d6f1700 1 Processor -- start 2024-01-15T11:25:09.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.051+0000 7ff92d6f1700 1 -- start start 2024-01-15T11:25:09.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.051+0000 7ff92d6f1700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff92806b9e0 0x7ff9280b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:09.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.051+0000 7ff92d6f1700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9280b1740 0x7ff9280c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:09.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.051+0000 7ff92d6f1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff92815bb30 0x7ff9280c6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:09.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.051+0000 7ff92d6f1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9280c6220 con 0x7ff92815bb30 2024-01-15T11:25:09.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.051+0000 7ff92d6f1700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9280cfe60 con 0x7ff9280b1740 2024-01-15T11:25:09.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.051+0000 7ff92d6f1700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9280d0140 con 0x7ff92806b9e0 2024-01-15T11:25:09.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.051+0000 7ff9277fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff92815bb30 0x7ff9280c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.051+0000 7ff9277fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff92815bb30 0x7ff9280c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36264/0 (socket says 172.21.15.113:36264) 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.051+0000 7ff9277fe700 1 -- 172.21.15.113:0/449607237 learned_addr learned my addr 172.21.15.113:0/449607237 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.051+0000 7ff9267fc700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9280b1740 0x7ff9280c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.051+0000 7ff926ffd700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff92806b9e0 0x7ff9280b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.052+0000 7ff9277fe700 1 -- 172.21.15.113:0/449607237 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff92806b9e0 msgr2=0x7ff9280b4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.052+0000 7ff9277fe700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff92806b9e0 0x7ff9280b4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.052+0000 7ff9277fe700 1 -- 172.21.15.113:0/449607237 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9280b1740 msgr2=0x7ff9280c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.052+0000 7ff9277fe700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9280b1740 0x7ff9280c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.052+0000 7ff9277fe700 1 -- 172.21.15.113:0/449607237 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c0094a0 con 0x7ff92815bb30 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.052+0000 7ff926ffd700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff92806b9e0 0x7ff9280b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.052+0000 7ff9277fe700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff92815bb30 0x7ff9280c6500 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff92000cc20 tx=0x7ff92000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.052+0000 7ff90f7fe700 1 -- 172.21.15.113:0/449607237 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff920003c00 con 0x7ff92815bb30 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.052+0000 7ff90f7fe700 1 -- 172.21.15.113:0/449607237 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff920003d60 con 0x7ff92815bb30 2024-01-15T11:25:09.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.052+0000 7ff92d6f1700 1 -- 172.21.15.113:0/449607237 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9280d0420 con 0x7ff92815bb30 2024-01-15T11:25:09.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.052+0000 7ff90f7fe700 1 -- 172.21.15.113:0/449607237 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92000ab90 con 0x7ff92815bb30 2024-01-15T11:25:09.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.052+0000 7ff92d6f1700 1 -- 172.21.15.113:0/449607237 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92815ab30 con 0x7ff92815bb30 2024-01-15T11:25:09.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.053+0000 7ff92d6f1700 1 -- 172.21.15.113:0/449607237 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9280cfcc0 con 0x7ff92815bb30 2024-01-15T11:25:09.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.053+0000 7ff90f7fe700 1 -- 172.21.15.113:0/449607237 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff92000acf0 con 0x7ff92815bb30 2024-01-15T11:25:09.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.054+0000 7ff90f7fe700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff91406b350 0x7ff91406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:09.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.054+0000 7ff90f7fe700 1 -- 172.21.15.113:0/449607237 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff9200899f0 con 0x7ff92815bb30 2024-01-15T11:25:09.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.054+0000 7ff926ffd700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff91406b350 0x7ff91406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:09.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.054+0000 7ff926ffd700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff91406b350 0x7ff91406d810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff92809fdb0 tx=0x7ff9180014f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:09.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.058+0000 7ff90f7fe700 1 -- 172.21.15.113:0/449607237 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9280cfcc0 con 0x7ff92815bb30 2024-01-15T11:25:09.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.247+0000 7ff92d6f1700 1 -- 172.21.15.113:0/449607237 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff9280cfcc0 con 0x7ff92815bb30 2024-01-15T11:25:09.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.247+0000 7ff90f7fe700 1 -- 172.21.15.113:0/449607237 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff9280cfcc0 con 0x7ff92815bb30 2024-01-15T11:25:09.246 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:09.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.249+0000 7ff92d6f1700 1 -- 172.21.15.113:0/449607237 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff91406b350 msgr2=0x7ff91406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:09.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.249+0000 7ff92d6f1700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff91406b350 0x7ff91406d810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff92809fdb0 tx=0x7ff9180014f0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.249+0000 7ff92d6f1700 1 -- 172.21.15.113:0/449607237 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff92815bb30 msgr2=0x7ff9280c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:09.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.249+0000 7ff92d6f1700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff92815bb30 0x7ff9280c6500 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff92000cc20 tx=0x7ff92000cf30 comp rx=0 tx=0).stop 2024-01-15T11:25:09.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.249+0000 7ff92d6f1700 1 -- 172.21.15.113:0/449607237 shutdown_connections 2024-01-15T11:25:09.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.249+0000 7ff92d6f1700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff91406b350 0x7ff91406d810 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.249+0000 7ff92d6f1700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff92815bb30 0x7ff9280c6500 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.249+0000 7ff92d6f1700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff92806b9e0 0x7ff9280b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.249+0000 7ff92d6f1700 1 --2- 172.21.15.113:0/449607237 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9280b1740 0x7ff9280c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.249+0000 7ff92d6f1700 1 -- 172.21.15.113:0/449607237 >> 172.21.15.113:0/449607237 conn(0x7ff92809aa00 msgr2=0x7ff9280d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:09.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.249+0000 7ff92d6f1700 1 -- 172.21.15.113:0/449607237 shutdown_connections 2024-01-15T11:25:09.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.249+0000 7ff92d6f1700 1 -- 172.21.15.113:0/449607237 wait complete. 2024-01-15T11:25:09.260 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T11:25:09.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.383+0000 7f3b7eab9700 1 Processor -- start 2024-01-15T11:25:09.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.383+0000 7f3b7eab9700 1 -- start start 2024-01-15T11:25:09.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.383+0000 7f3b7eab9700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3b7815afc0 0x7f3b780b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:09.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.383+0000 7f3b7eab9700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3b780b1b10 0x7f3b780b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:09.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.383+0000 7f3b7eab9700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b780af8c0 con 0x7f3b780cf860 2024-01-15T11:25:09.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.383+0000 7f3b7eab9700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b780b0790 con 0x7f3b780b1b10 2024-01-15T11:25:09.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.383+0000 7f3b7eab9700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b780afb10 con 0x7f3b7815afc0 2024-01-15T11:25:09.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.383+0000 7f3b7d056700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3b780b1b10 0x7f3b780b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:09.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.384+0000 7f3b7d056700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3b780b1b10 0x7f3b780b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36280/0 (socket says 172.21.15.113:36280) 2024-01-15T11:25:09.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.384+0000 7f3b7d056700 1 -- 172.21.15.113:0/734513250 learned_addr learned my addr 172.21.15.113:0/734513250 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:09.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.384+0000 7f3b7d056700 1 -- 172.21.15.113:0/734513250 >> v2:172.21.15.141:3300/0 conn(0x7f3b7815afc0 msgr2=0x7f3b780b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:09.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.384+0000 7f3b7d056700 1 --2- 172.21.15.113:0/734513250 >> v2:172.21.15.141:3300/0 conn(0x7f3b7815afc0 0x7f3b780b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.384+0000 7f3b7d056700 1 -- 172.21.15.113:0/734513250 >> v1:172.21.15.141:6789/0 conn(0x7f3b780cf860 legacy=0x7f3b780b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:09.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.384+0000 7f3b7d056700 1 -- 172.21.15.113:0/734513250 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b780afdf0 con 0x7f3b780b1b10 2024-01-15T11:25:09.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.384+0000 7f3b7d056700 1 --2- 172.21.15.113:0/734513250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b780b1b10 0x7f3b780b3290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b7806fc60 tx=0x7f3b7000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e724e310b43028 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:09.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.384+0000 7f3b777fe700 1 -- 172.21.15.113:0/734513250 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b70010070 con 0x7f3b780b1b10 2024-01-15T11:25:09.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.384+0000 7f3b777fe700 1 -- 172.21.15.113:0/734513250 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7000dd50 con 0x7f3b780b1b10 2024-01-15T11:25:09.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.384+0000 7f3b777fe700 1 -- 172.21.15.113:0/734513250 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7000b310 con 0x7f3b780b1b10 2024-01-15T11:25:09.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.385+0000 7f3b7eab9700 1 -- 172.21.15.113:0/734513250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b780b1b10 msgr2=0x7f3b780b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:09.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.385+0000 7f3b7eab9700 1 --2- 172.21.15.113:0/734513250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b780b1b10 0x7f3b780b3290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b7806fc60 tx=0x7f3b7000d430 comp rx=0 tx=0).stop 2024-01-15T11:25:09.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.385+0000 7f3b7eab9700 1 -- 172.21.15.113:0/734513250 shutdown_connections 2024-01-15T11:25:09.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.385+0000 7f3b7eab9700 1 --2- 172.21.15.113:0/734513250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b780b1b10 0x7f3b780b3290 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.385+0000 7f3b7eab9700 1 --2- 172.21.15.113:0/734513250 >> v2:172.21.15.141:3300/0 conn(0x7f3b7815afc0 0x7f3b780b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.385+0000 7f3b7eab9700 1 -- 172.21.15.113:0/734513250 >> 172.21.15.113:0/734513250 conn(0x7f3b780b57b0 msgr2=0x7f3b780b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:09.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.385+0000 7f3b7eab9700 1 -- 172.21.15.113:0/734513250 shutdown_connections 2024-01-15T11:25:09.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.385+0000 7f3b7eab9700 1 -- 172.21.15.113:0/734513250 wait complete. 2024-01-15T11:25:09.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.386+0000 7f3b7eab9700 1 Processor -- start 2024-01-15T11:25:09.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.386+0000 7f3b7eab9700 1 -- start start 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7eab9700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b780b1b10 0x7f3b7814d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7eab9700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b780cf860 0x7f3b780adc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7eab9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b7815afc0 0x7f3b7815d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7eab9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7814de70 con 0x7f3b7815afc0 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7eab9700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b780af1b0 con 0x7f3b780b1b10 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7eab9700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b780aebf0 con 0x7f3b780cf860 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7d056700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b7815afc0 0x7f3b7815d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7d056700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b7815afc0 0x7f3b7815d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36292/0 (socket says 172.21.15.113:36292) 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7d056700 1 -- 172.21.15.113:0/1360503336 learned_addr learned my addr 172.21.15.113:0/1360503336 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7c855700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b780b1b10 0x7f3b7814d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b77fff700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b780cf860 0x7f3b780adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7d056700 1 -- 172.21.15.113:0/1360503336 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b780cf860 msgr2=0x7f3b780adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:09.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7d056700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b780cf860 0x7f3b780adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7d056700 1 -- 172.21.15.113:0/1360503336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b780b1b10 msgr2=0x7f3b7814d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7d056700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b780b1b10 0x7f3b7814d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7d056700 1 -- 172.21.15.113:0/1360503336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7000bd40 con 0x7f3b7815afc0 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7c855700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b780b1b10 0x7f3b7814d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b77fff700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b780cf860 0x7f3b780adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.387+0000 7f3b7d056700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b7815afc0 0x7f3b7815d620 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b70004000 tx=0x7f3b70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.388+0000 7f3b757fa700 1 -- 172.21.15.113:0/1360503336 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7000ab20 con 0x7f3b7815afc0 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.388+0000 7f3b757fa700 1 -- 172.21.15.113:0/1360503336 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7000ac80 con 0x7f3b7815afc0 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.388+0000 7f3b7eab9700 1 -- 172.21.15.113:0/1360503336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b780aeed0 con 0x7f3b7815afc0 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.388+0000 7f3b757fa700 1 -- 172.21.15.113:0/1360503336 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b70004030 con 0x7f3b7815afc0 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.388+0000 7f3b7eab9700 1 -- 172.21.15.113:0/1360503336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b7815d100 con 0x7f3b7815afc0 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.388+0000 7f3b7eab9700 1 -- 172.21.15.113:0/1360503336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b7815c520 con 0x7f3b7815afc0 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.389+0000 7f3b757fa700 1 -- 172.21.15.113:0/1360503336 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3b7001b050 con 0x7f3b7815afc0 2024-01-15T11:25:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.389+0000 7f3b757fa700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3b5c06b350 0x7f3b5c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:09.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.389+0000 7f3b757fa700 1 -- 172.21.15.113:0/1360503336 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3b7008ce20 con 0x7f3b7815afc0 2024-01-15T11:25:09.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.389+0000 7f3b7c855700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3b5c06b350 0x7f3b5c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:09.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.390+0000 7f3b7c855700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3b5c06b350 0x7f3b5c06d810 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00a8f0 tx=0x7f3b6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:09.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.394+0000 7f3b757fa700 1 -- 172.21.15.113:0/1360503336 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b70058070 con 0x7f3b7815afc0 2024-01-15T11:25:09.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.629+0000 7f3b7eab9700 1 -- 172.21.15.113:0/1360503336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b78067f60 con 0x7f3b7815afc0 2024-01-15T11:25:09.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.629+0000 7f3b757fa700 1 -- 172.21.15.113:0/1360503336 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f3b70019540 con 0x7f3b7815afc0 2024-01-15T11:25:09.628 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:09.628 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/891522731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":891522731},{"type":"v1","addr":"172.21.15.113:6837","nonce":891522731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-01-15T11:25:09.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.631+0000 7f3b7eab9700 1 -- 172.21.15.113:0/1360503336 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3b5c06b350 msgr2=0x7f3b5c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:09.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.631+0000 7f3b7eab9700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3b5c06b350 0x7f3b5c06d810 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00a8f0 tx=0x7f3b6c008040 comp rx=0 tx=0).stop 2024-01-15T11:25:09.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.631+0000 7f3b7eab9700 1 -- 172.21.15.113:0/1360503336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b7815afc0 msgr2=0x7f3b7815d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:09.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.631+0000 7f3b7eab9700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b7815afc0 0x7f3b7815d620 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b70004000 tx=0x7f3b70004ad0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.631+0000 7f3b7eab9700 1 -- 172.21.15.113:0/1360503336 shutdown_connections 2024-01-15T11:25:09.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.631+0000 7f3b7eab9700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3b5c06b350 0x7f3b5c06d810 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.631+0000 7f3b7eab9700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b7815afc0 0x7f3b7815d620 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.631+0000 7f3b7eab9700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b780b1b10 0x7f3b7814d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.631+0000 7f3b7eab9700 1 --2- 172.21.15.113:0/1360503336 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b780cf860 0x7f3b780adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:09.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.631+0000 7f3b7eab9700 1 -- 172.21.15.113:0/1360503336 >> 172.21.15.113:0/1360503336 conn(0x7f3b780b57b0 msgr2=0x7f3b7806d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:09.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.632+0000 7f3b7eab9700 1 -- 172.21.15.113:0/1360503336 shutdown_connections 2024-01-15T11:25:09.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:09.632+0000 7f3b7eab9700 1 -- 172.21.15.113:0/1360503336 wait complete. 2024-01-15T11:25:09.631 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 2 2024-01-15T11:25:11.642 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T11:25:11.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.777+0000 7ff846ee8700 1 Processor -- start 2024-01-15T11:25:11.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.777+0000 7ff846ee8700 1 -- start start 2024-01-15T11:25:11.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff846ee8700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff8400ce4d0 0x7ff8400d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:11.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff846ee8700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 0x7ff8400d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:11.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff846ee8700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8400d0150 con 0x7ff84015e080 2024-01-15T11:25:11.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff846ee8700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84015e460 con 0x7ff8400ce4d0 2024-01-15T11:25:11.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff846ee8700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400cc9b0 con 0x7ff8400ce910 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff845485700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 0x7ff8400d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff83ffff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff8400ce4d0 0x7ff8400d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff845485700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 0x7ff8400d2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:46376/0 (socket says 172.21.15.113:46376) 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff845485700 1 -- 172.21.15.113:0/1375107803 learned_addr learned my addr 172.21.15.113:0/1375107803 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff844c84700 1 --1- 172.21.15.113:0/1375107803 >> v1:172.21.15.141:6789/0 conn(0x7ff84015e080 0x7ff8400cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:40746/0 (socket says 172.21.15.113:40746) 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff845485700 1 -- 172.21.15.113:0/1375107803 >> v2:172.21.15.141:3300/0 conn(0x7ff8400ce4d0 msgr2=0x7ff8400d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff845485700 1 --2- 172.21.15.113:0/1375107803 >> v2:172.21.15.141:3300/0 conn(0x7ff8400ce4d0 0x7ff8400d1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff845485700 1 -- 172.21.15.113:0/1375107803 >> v1:172.21.15.141:6789/0 conn(0x7ff84015e080 legacy=0x7ff8400cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff845485700 1 -- 172.21.15.113:0/1375107803 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8400cee50 con 0x7ff8400ce910 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.778+0000 7ff83ffff700 1 --2- 172.21.15.113:0/1375107803 >> v2:172.21.15.141:3300/0 conn(0x7ff8400ce4d0 0x7ff8400d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.779+0000 7ff845485700 1 --2- 172.21.15.113:0/1375107803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 0x7ff8400d2120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff838009e50 tx=0x7ff83800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4d5da09b466eb93 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.779+0000 7ff83f7fe700 1 -- 172.21.15.113:0/1375107803 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83800e070 con 0x7ff8400ce910 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.779+0000 7ff83f7fe700 1 -- 172.21.15.113:0/1375107803 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83800b910 con 0x7ff8400ce910 2024-01-15T11:25:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.779+0000 7ff83f7fe700 1 -- 172.21.15.113:0/1375107803 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83800bcc0 con 0x7ff8400ce910 2024-01-15T11:25:11.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.779+0000 7ff846ee8700 1 -- 172.21.15.113:0/1375107803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 msgr2=0x7ff8400d2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:11.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.779+0000 7ff846ee8700 1 --2- 172.21.15.113:0/1375107803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 0x7ff8400d2120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff838009e50 tx=0x7ff83800b330 comp rx=0 tx=0).stop 2024-01-15T11:25:11.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.779+0000 7ff846ee8700 1 -- 172.21.15.113:0/1375107803 shutdown_connections 2024-01-15T11:25:11.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.779+0000 7ff846ee8700 1 --2- 172.21.15.113:0/1375107803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 0x7ff8400d2120 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:11.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.779+0000 7ff846ee8700 1 --2- 172.21.15.113:0/1375107803 >> v2:172.21.15.141:3300/0 conn(0x7ff8400ce4d0 0x7ff8400d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:11.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.779+0000 7ff846ee8700 1 -- 172.21.15.113:0/1375107803 >> 172.21.15.113:0/1375107803 conn(0x7ff840073130 msgr2=0x7ff84009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:11.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.779+0000 7ff846ee8700 1 -- 172.21.15.113:0/1375107803 shutdown_connections 2024-01-15T11:25:11.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.779+0000 7ff846ee8700 1 -- 172.21.15.113:0/1375107803 wait complete. 2024-01-15T11:25:11.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.781+0000 7ff846ee8700 1 Processor -- start 2024-01-15T11:25:11.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.781+0000 7ff846ee8700 1 -- start start 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.781+0000 7ff846ee8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8400ce4d0 0x7ff8401f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.781+0000 7ff846ee8700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 0x7ff8400b6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.781+0000 7ff846ee8700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff84015e080 0x7ff8400b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.781+0000 7ff846ee8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400b6150 con 0x7ff8400ce4d0 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.781+0000 7ff846ee8700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400b5280 con 0x7ff84015e080 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.781+0000 7ff846ee8700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400b54d0 con 0x7ff8400ce910 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff844c84700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8400ce4d0 0x7ff8401f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff844c84700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8400ce4d0 0x7ff8401f6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36306/0 (socket says 172.21.15.113:36306) 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff844c84700 1 -- 172.21.15.113:0/44141635 learned_addr learned my addr 172.21.15.113:0/44141635 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff845485700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff84015e080 0x7ff8400b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff83ffff700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 0x7ff8400b6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff844c84700 1 -- 172.21.15.113:0/44141635 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 msgr2=0x7ff8400b6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff844c84700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 0x7ff8400b6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:11.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff844c84700 1 -- 172.21.15.113:0/44141635 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff84015e080 msgr2=0x7ff8400b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff844c84700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff84015e080 0x7ff8400b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff844c84700 1 -- 172.21.15.113:0/44141635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c0094a0 con 0x7ff8400ce4d0 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff83ffff700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 0x7ff8400b6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff844c84700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8400ce4d0 0x7ff8401f6dd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff83400c500 tx=0x7ff83400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff83d7fa700 1 -- 172.21.15.113:0/44141635 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff834005cb0 con 0x7ff8400ce4d0 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff83d7fa700 1 -- 172.21.15.113:0/44141635 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff834005e10 con 0x7ff8400ce4d0 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff846ee8700 1 -- 172.21.15.113:0/44141635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff838009b30 con 0x7ff8400ce4d0 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff83d7fa700 1 -- 172.21.15.113:0/44141635 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8340072d0 con 0x7ff8400ce4d0 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.782+0000 7ff846ee8700 1 -- 172.21.15.113:0/44141635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8400b5a90 con 0x7ff8400ce4d0 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.783+0000 7ff846ee8700 1 -- 172.21.15.113:0/44141635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff840067f60 con 0x7ff8400ce4d0 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.784+0000 7ff83d7fa700 1 -- 172.21.15.113:0/44141635 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff834007430 con 0x7ff8400ce4d0 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.784+0000 7ff83d7fa700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff82406b350 0x7ff82406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.784+0000 7ff83d7fa700 1 -- 172.21.15.113:0/44141635 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff834088a00 con 0x7ff8400ce4d0 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.784+0000 7ff83ffff700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff82406b350 0x7ff82406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:11.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.784+0000 7ff83ffff700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff82406b350 0x7ff82406d810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff8400ca170 tx=0x7ff82c009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:11.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:11.789+0000 7ff83d7fa700 1 -- 172.21.15.113:0/44141635 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff834053c70 con 0x7ff8400ce4d0 2024-01-15T11:25:12.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.019+0000 7ff846ee8700 1 -- 172.21.15.113:0/44141635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8400d19c0 con 0x7ff8400ce4d0 2024-01-15T11:25:12.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.019+0000 7ff83d7fa700 1 -- 172.21.15.113:0/44141635 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3703 (secure 0 0 0) 0x7ff834017020 con 0x7ff8400ce4d0 2024-01-15T11:25:12.018 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:12.018 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3164513953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":3164513953},{"type":"v1","addr":"172.21.15.113:6835","nonce":3164513953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/891522731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":891522731},{"type":"v1","addr":"172.21.15.113:6837","nonce":891522731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2158560535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2158560535},{"type":"v1","addr":"172.21.15.141:6834","nonce":2158560535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/1012738257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1012738257},{"type":"v1","addr":"172.21.15.141:6835","nonce":1012738257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-01-15T11:25:12.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.021+0000 7ff846ee8700 1 -- 172.21.15.113:0/44141635 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff82406b350 msgr2=0x7ff82406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.021+0000 7ff846ee8700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff82406b350 0x7ff82406d810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff8400ca170 tx=0x7ff82c009210 comp rx=0 tx=0).stop 2024-01-15T11:25:12.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.021+0000 7ff846ee8700 1 -- 172.21.15.113:0/44141635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8400ce4d0 msgr2=0x7ff8401f6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.021+0000 7ff846ee8700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8400ce4d0 0x7ff8401f6dd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff83400c500 tx=0x7ff83400c810 comp rx=0 tx=0).stop 2024-01-15T11:25:12.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.021+0000 7ff846ee8700 1 -- 172.21.15.113:0/44141635 shutdown_connections 2024-01-15T11:25:12.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.021+0000 7ff846ee8700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff82406b350 0x7ff82406d810 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.021+0000 7ff846ee8700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff84015e080 0x7ff8400b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.021+0000 7ff846ee8700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8400ce4d0 0x7ff8401f6dd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.021+0000 7ff846ee8700 1 --2- 172.21.15.113:0/44141635 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff8400ce910 0x7ff8400b6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.021+0000 7ff846ee8700 1 -- 172.21.15.113:0/44141635 >> 172.21.15.113:0/44141635 conn(0x7ff840073130 msgr2=0x7ff84009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:12.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.021+0000 7ff846ee8700 1 -- 172.21.15.113:0/44141635 shutdown_connections 2024-01-15T11:25:12.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.021+0000 7ff846ee8700 1 -- 172.21.15.113:0/44141635 wait complete. 2024-01-15T11:25:12.021 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 3 2024-01-15T11:25:12.032 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:12.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.156+0000 7fbbc2ff5700 1 Processor -- start 2024-01-15T11:25:12.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.156+0000 7fbbc2ff5700 1 -- start start 2024-01-15T11:25:12.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbc2ff5700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbbc15e480 0x7fbbbc0d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbc2ff5700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbbc0d2a70 con 0x7fbbbc0b6060 2024-01-15T11:25:12.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbc2ff5700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbbc0d2d50 con 0x7fbbbc0d3610 2024-01-15T11:25:12.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbc2ff5700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc0d3030 con 0x7fbbbc15e480 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbc1592700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fbbbc0b6060 0x7fbbbc0b7860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:44248/0 (socket says 172.21.15.113:44248) 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbc1592700 1 -- 172.21.15.113:0/2237764602 learned_addr learned my addr 172.21.15.113:0/2237764602 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbc0d91700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbbc15e480 0x7fbbbc0d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009605449 0 0) 0x7fbbbc0d2a70 con 0x7fbbbc0b6060 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbac0035b0 con 0x7fbbbc0b6060 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559623225 0 0) 0x7fbbbc0d2d50 con 0x7fbbbc0d3610 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbbc0d2a70 con 0x7fbbbc0d3610 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1559632329 0 0) 0x7fbbac0035b0 con 0x7fbbbc0b6060 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbbc15e480 msgr2=0x7fbbbc0d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbbb7fe700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbbc15e480 0x7fbbbc0d2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 >> v1:172.21.15.141:6789/0 conn(0x7fbbbc0d3610 legacy=0x7fbbbc0b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc0b7340 con 0x7fbbbc0b6060 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbc0d91700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbbc15e480 0x7fbbbc0d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.157+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fbbb40019f0 con 0x7fbbbc0b6060 2024-01-15T11:25:12.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 >> v1:172.21.15.113:6789/0 conn(0x7fbbbc0b6060 legacy=0x7fbbbc0b7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbbb7fe700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbac003bc0 0x7fbbac003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbbb7fe700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbbc15e480 0x7fbbac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbbb7fe700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbbac004d90 0x7fbbac005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc0d3030 con 0x7fbbac003bc0 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac00baf0 con 0x7fbbac004d90 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac00bcb0 con 0x7fbbbc15e480 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbc0d91700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbac003bc0 0x7fbbac003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbbbfff700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbbc15e480 0x7fbbac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbc1592700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbbac004d90 0x7fbbac005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbc0d91700 1 -- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbbc15e480 msgr2=0x7fbbac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbc0d91700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbbc15e480 0x7fbbac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbc0d91700 1 -- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbbac004d90 msgr2=0x7fbbac005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbc0d91700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbbac004d90 0x7fbbac005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbc0d91700 1 -- 172.21.15.113:0/2237764602 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb00094a0 con 0x7fbbac003bc0 2024-01-15T11:25:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbbbfff700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbbc15e480 0x7fbbac004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbc0d91700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbac003bc0 0x7fbbac003fc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009d30 tx=0x7fbbac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b8922810256087 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0009e80 con 0x7fbbac003bc0 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb000e9a0 con 0x7fbbac003bc0 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.158+0000 7fbbbb7fe700 1 -- 172.21.15.113:0/2237764602 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb000ed10 con 0x7fbbac003bc0 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.159+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/2237764602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbac003bc0 msgr2=0x7fbbac003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.159+0000 7fbbc2ff5700 1 --2- 172.21.15.113:0/2237764602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbac003bc0 0x7fbbac003fc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009d30 tx=0x7fbbac005880 comp rx=0 tx=0).stop 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.159+0000 7fbbc1592700 1 -- 172.21.15.113:0/2237764602 reap_dead start 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.159+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/2237764602 shutdown_connections 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.159+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/2237764602 >> 172.21.15.113:0/2237764602 conn(0x7fbbbc0d0950 msgr2=0x7fbbbc0d0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.159+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/2237764602 shutdown_connections 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.159+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/2237764602 wait complete. 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.160+0000 7fbbc2ff5700 1 Processor -- start 2024-01-15T11:25:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.160+0000 7fbbc2ff5700 1 -- start start 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbc2ff5700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbac003bc0 0x7fbbbc1612f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbc2ff5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbbc157d90 0x7fbbbc160a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbc2ff5700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbbbc159790 0x7fbbbc159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbc2ff5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc161830 con 0x7fbbbc157d90 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbc2ff5700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc158270 con 0x7fbbbc159790 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbc2ff5700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc152f10 con 0x7fbbac003bc0 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbbbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbbc157d90 0x7fbbbc160a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbbbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbbc157d90 0x7fbbbc160a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36328/0 (socket says 172.21.15.113:36328) 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbbbfff700 1 -- 172.21.15.113:0/3917083898 learned_addr learned my addr 172.21.15.113:0/3917083898 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbc1592700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbbbc159790 0x7fbbbc159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbc0d91700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbac003bc0 0x7fbbbc1612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbbbfff700 1 -- 172.21.15.113:0/3917083898 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbac003bc0 msgr2=0x7fbbbc1612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbbbfff700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbac003bc0 0x7fbbbc1612f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbbbfff700 1 -- 172.21.15.113:0/3917083898 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbbbc159790 msgr2=0x7fbbbc159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbbbfff700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbbbc159790 0x7fbbbc159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbbbfff700 1 -- 172.21.15.113:0/3917083898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc1f81f0 con 0x7fbbbc157d90 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbbbfff700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbbc157d90 0x7fbbbc160a50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbba800e7b0 tx=0x7fbba800eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbb97fa700 1 -- 172.21.15.113:0/3917083898 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba8017070 con 0x7fbbbc157d90 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbb97fa700 1 -- 172.21.15.113:0/3917083898 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba80052e0 con 0x7fbbbc157d90 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/3917083898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbbc1f84d0 con 0x7fbbbc157d90 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbb97fa700 1 -- 172.21.15.113:0/3917083898 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba8005450 con 0x7fbbbc157d90 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.161+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/3917083898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbbc1576d0 con 0x7fbbbc157d90 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.162+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/3917083898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbbc0751e0 con 0x7fbbbc157d90 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.163+0000 7fbbb97fa700 1 -- 172.21.15.113:0/3917083898 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbba8006860 con 0x7fbbbc157d90 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.163+0000 7fbbb97fa700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fbba006b350 0x7fbba006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.163+0000 7fbbb97fa700 1 -- 172.21.15.113:0/3917083898 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbba8088f10 con 0x7fbbbc157d90 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.163+0000 7fbbc0d91700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fbba006b350 0x7fbba006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.163+0000 7fbbc0d91700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fbba006b350 0x7fbba006d810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00090b0 tx=0x7fbbb0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:12.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.167+0000 7fbbb97fa700 1 -- 172.21.15.113:0/3917083898 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbba8054160 con 0x7fbbbc157d90 2024-01-15T11:25:12.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.361+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/3917083898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbbc0b6810 con 0x7fbbbc157d90 2024-01-15T11:25:12.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.361+0000 7fbbb97fa700 1 -- 172.21.15.113:0/3917083898 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12910 (secure 0 0 0) 0x7fbba8057780 con 0x7fbbbc157d90 2024-01-15T11:25:12.360 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:12.360 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":16,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:04.497258+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:12.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.364+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/3917083898 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fbba006b350 msgr2=0x7fbba006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.364+0000 7fbbc2ff5700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fbba006b350 0x7fbba006d810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00090b0 tx=0x7fbbb0008030 comp rx=0 tx=0).stop 2024-01-15T11:25:12.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.364+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/3917083898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbbc157d90 msgr2=0x7fbbbc160a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.364+0000 7fbbc2ff5700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbbc157d90 0x7fbbbc160a50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbba800e7b0 tx=0x7fbba800eb70 comp rx=0 tx=0).stop 2024-01-15T11:25:12.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.364+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/3917083898 shutdown_connections 2024-01-15T11:25:12.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.364+0000 7fbbc2ff5700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fbba006b350 0x7fbba006d810 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.364+0000 7fbbc2ff5700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbbbc159790 0x7fbbbc159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.364+0000 7fbbc2ff5700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbbac003bc0 0x7fbbbc1612f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.364+0000 7fbbc2ff5700 1 --2- 172.21.15.113:0/3917083898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbbc157d90 0x7fbbbc160a50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.364+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/3917083898 >> 172.21.15.113:0/3917083898 conn(0x7fbbbc0d0950 msgr2=0x7fbbbc0cf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:12.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.364+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/3917083898 shutdown_connections 2024-01-15T11:25:12.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.364+0000 7fbbc2ff5700 1 -- 172.21.15.113:0/3917083898 wait complete. 2024-01-15T11:25:12.374 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:12.374 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:12.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.499+0000 7f1103313700 1 Processor -- start 2024-01-15T11:25:12.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.500+0000 7f1103313700 1 -- start start 2024-01-15T11:25:12.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f1103313700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f10fc158600 0x7f10fc06f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f1103313700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10fc06e4e0 con 0x7f10fc15ee20 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f1103313700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10fc06e7c0 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f1103313700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10fc06fac0 con 0x7f10fc158600 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f11010af700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f10fc158600 0x7f10fc06f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f11018b0700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f10fc15ee20 0x7f10fc06aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:44254/0 (socket says 172.21.15.113:44254) 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f11018b0700 1 -- 172.21.15.113:0/3369204794 learned_addr learned my addr 172.21.15.113:0/3369204794 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f11010af700 1 -- 172.21.15.113:0/3369204794 >> v1:172.21.15.141:6789/0 conn(0x7f10fc0b4ef0 legacy=0x7f10fc15e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f11010af700 1 -- 172.21.15.113:0/3369204794 >> v1:172.21.15.113:6789/0 conn(0x7f10fc15ee20 legacy=0x7f10fc06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f10effff700 1 -- 172.21.15.113:0/3369204794 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966879653 0 0) 0x7f10fc06e4e0 con 0x7f10fc15ee20 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f11010af700 1 -- 172.21.15.113:0/3369204794 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc06eaa0 con 0x7f10fc158600 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f11010af700 1 --2- 172.21.15.113:0/3369204794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10fc158600 0x7f10fc06f580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009870 tx=0x7f10f0009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf3768bea0cb68 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f10effff700 1 -- 172.21.15.113:0/3369204794 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0011830 con 0x7f10fc158600 2024-01-15T11:25:12.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f10effff700 1 -- 172.21.15.113:0/3369204794 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f0011990 con 0x7f10fc158600 2024-01-15T11:25:12.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.501+0000 7f10effff700 1 -- 172.21.15.113:0/3369204794 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0011d20 con 0x7f10fc158600 2024-01-15T11:25:12.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.502+0000 7f1103313700 1 -- 172.21.15.113:0/3369204794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10fc158600 msgr2=0x7f10fc06f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.502+0000 7f1103313700 1 --2- 172.21.15.113:0/3369204794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10fc158600 0x7f10fc06f580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009870 tx=0x7f10f0009b80 comp rx=0 tx=0).stop 2024-01-15T11:25:12.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.502+0000 7f1103313700 1 -- 172.21.15.113:0/3369204794 shutdown_connections 2024-01-15T11:25:12.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.502+0000 7f1103313700 1 --2- 172.21.15.113:0/3369204794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10fc158600 0x7f10fc06f580 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.502+0000 7f1103313700 1 -- 172.21.15.113:0/3369204794 >> 172.21.15.113:0/3369204794 conn(0x7f10fc0b3250 msgr2=0x7f10fc0d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:12.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.502+0000 7f1103313700 1 -- 172.21.15.113:0/3369204794 shutdown_connections 2024-01-15T11:25:12.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.502+0000 7f1103313700 1 -- 172.21.15.113:0/3369204794 wait complete. 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.503+0000 7f1103313700 1 Processor -- start 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.503+0000 7f1103313700 1 -- start start 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f1103313700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10fc0b4ef0 0x7f10fc1f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f1103313700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10fc158600 0x7f10fc1f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f1103313700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10fc15ee20 0x7f10fc151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f1103313700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc1f3330 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f1103313700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc1f3090 con 0x7f10fc158600 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f1103313700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc1f51a0 con 0x7f10fc15ee20 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11010af700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10fc0b4ef0 0x7f10fc1f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11010af700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10fc0b4ef0 0x7f10fc1f1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36354/0 (socket says 172.21.15.113:36354) 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11010af700 1 -- 172.21.15.113:0/2862479837 learned_addr learned my addr 172.21.15.113:0/2862479837 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11008ae700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10fc158600 0x7f10fc1f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11018b0700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10fc15ee20 0x7f10fc151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11010af700 1 -- 172.21.15.113:0/2862479837 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10fc15ee20 msgr2=0x7f10fc151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11010af700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10fc15ee20 0x7f10fc151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11010af700 1 -- 172.21.15.113:0/2862479837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10fc158600 msgr2=0x7f10fc1f4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11010af700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10fc158600 0x7f10fc1f4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11010af700 1 -- 172.21.15.113:0/2862479837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f00094a0 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11008ae700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10fc158600 0x7f10fc1f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11018b0700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10fc15ee20 0x7f10fc151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f11010af700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10fc0b4ef0 0x7f10fc1f1ec0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f10f0000c00 tx=0x7f10f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f10edffb700 1 -- 172.21.15.113:0/2862479837 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f00045b0 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f10edffb700 1 -- 172.21.15.113:0/2862479837 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f001f070 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f1103313700 1 -- 172.21.15.113:0/2862479837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10fc1f2510 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.504+0000 7f10edffb700 1 -- 172.21.15.113:0/2862479837 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0003b00 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.505+0000 7f1103313700 1 -- 172.21.15.113:0/2862479837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10fc1f2ad0 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.505+0000 7f1103313700 1 -- 172.21.15.113:0/2862479837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10fc15e260 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.506+0000 7f10edffb700 1 -- 172.21.15.113:0/2862479837 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f10f0003c60 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.506+0000 7f10edffb700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f10e006b350 0x7f10e006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.506+0000 7f10edffb700 1 -- 172.21.15.113:0/2862479837 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f10f0021070 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.506+0000 7f11008ae700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f10e006b350 0x7f10e006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.507+0000 7f11008ae700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f10e006b350 0x7f10e006d810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f10fc09e440 tx=0x7f10f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:12.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.511+0000 7f10edffb700 1 -- 172.21.15.113:0/2862479837 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10f0056650 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.702+0000 7f1103313700 1 -- 172.21.15.113:0/2862479837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10fc15e4b0 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.702+0000 7f10edffb700 1 -- 172.21.15.113:0/2862479837 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f10f002b0a0 con 0x7f10fc0b4ef0 2024-01-15T11:25:12.701 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.704+0000 7f1103313700 1 -- 172.21.15.113:0/2862479837 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f10e006b350 msgr2=0x7f10e006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.704+0000 7f1103313700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f10e006b350 0x7f10e006d810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f10fc09e440 tx=0x7f10f8009040 comp rx=0 tx=0).stop 2024-01-15T11:25:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.704+0000 7f1103313700 1 -- 172.21.15.113:0/2862479837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10fc0b4ef0 msgr2=0x7f10fc1f1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.704+0000 7f1103313700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10fc0b4ef0 0x7f10fc1f1ec0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f10f0000c00 tx=0x7f10f0004750 comp rx=0 tx=0).stop 2024-01-15T11:25:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.704+0000 7f1103313700 1 -- 172.21.15.113:0/2862479837 shutdown_connections 2024-01-15T11:25:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.704+0000 7f1103313700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f10e006b350 0x7f10e006d810 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.704+0000 7f1103313700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10fc15ee20 0x7f10fc151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.704+0000 7f1103313700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10fc0b4ef0 0x7f10fc1f1ec0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.704+0000 7f1103313700 1 --2- 172.21.15.113:0/2862479837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10fc158600 0x7f10fc1f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.704+0000 7f1103313700 1 -- 172.21.15.113:0/2862479837 >> 172.21.15.113:0/2862479837 conn(0x7f10fc0b3250 msgr2=0x7f10fc0b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:12.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.704+0000 7f1103313700 1 -- 172.21.15.113:0/2862479837 shutdown_connections 2024-01-15T11:25:12.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.704+0000 7f1103313700 1 -- 172.21.15.113:0/2862479837 wait complete. 2024-01-15T11:25:12.715 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T11:25:12.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.836+0000 7f862b4ad700 1 Processor -- start 2024-01-15T11:25:12.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.836+0000 7f862b4ad700 1 -- start start 2024-01-15T11:25:12.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f862b4ad700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8624073f90 0x7f86240ce090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f862b4ad700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f862415c140 0x7f86240cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f862b4ad700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8624074e90 con 0x7f862406ee50 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f862b4ad700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86240ce5d0 con 0x7f8624073f90 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f862b4ad700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862415c700 con 0x7f862415c140 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f8629249700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8624073f90 0x7f86240ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f8629249700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8624073f90 0x7f86240ce090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36360/0 (socket says 172.21.15.113:36360) 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f8629249700 1 -- 172.21.15.113:0/4010544759 learned_addr learned my addr 172.21.15.113:0/4010544759 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f8629a4a700 1 --2- 172.21.15.113:0/4010544759 >> v2:172.21.15.141:3300/0 conn(0x7f862415c140 0x7f86240cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f8629249700 1 -- 172.21.15.113:0/4010544759 >> v2:172.21.15.141:3300/0 conn(0x7f862415c140 msgr2=0x7f86240cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f8629249700 1 --2- 172.21.15.113:0/4010544759 >> v2:172.21.15.141:3300/0 conn(0x7f862415c140 0x7f86240cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f8629249700 1 -- 172.21.15.113:0/4010544759 >> v1:172.21.15.141:6789/0 conn(0x7f862406ee50 legacy=0x7f8624074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f8629249700 1 -- 172.21.15.113:0/4010544759 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862415c9e0 con 0x7f8624073f90 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f8629249700 1 --2- 172.21.15.113:0/4010544759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8624073f90 0x7f86240ce090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f86200098f0 tx=0x7f8620009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e205344a019e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f861bfff700 1 -- 172.21.15.113:0/4010544759 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620004230 con 0x7f8624073f90 2024-01-15T11:25:12.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f861bfff700 1 -- 172.21.15.113:0/4010544759 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8620004390 con 0x7f8624073f90 2024-01-15T11:25:12.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.837+0000 7f861bfff700 1 -- 172.21.15.113:0/4010544759 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86200036a0 con 0x7f8624073f90 2024-01-15T11:25:12.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.838+0000 7f862b4ad700 1 -- 172.21.15.113:0/4010544759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8624073f90 msgr2=0x7f86240ce090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.838+0000 7f862b4ad700 1 --2- 172.21.15.113:0/4010544759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8624073f90 0x7f86240ce090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f86200098f0 tx=0x7f8620009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:12.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.838+0000 7f862b4ad700 1 -- 172.21.15.113:0/4010544759 shutdown_connections 2024-01-15T11:25:12.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.838+0000 7f862b4ad700 1 --2- 172.21.15.113:0/4010544759 >> v2:172.21.15.141:3300/0 conn(0x7f862415c140 0x7f86240cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.838+0000 7f862b4ad700 1 --2- 172.21.15.113:0/4010544759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8624073f90 0x7f86240ce090 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.838+0000 7f862b4ad700 1 -- 172.21.15.113:0/4010544759 >> 172.21.15.113:0/4010544759 conn(0x7f86240cd570 msgr2=0x7f86240cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:12.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.838+0000 7f862b4ad700 1 -- 172.21.15.113:0/4010544759 shutdown_connections 2024-01-15T11:25:12.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.838+0000 7f862b4ad700 1 -- 172.21.15.113:0/4010544759 wait complete. 2024-01-15T11:25:12.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.839+0000 7f862b4ad700 1 Processor -- start 2024-01-15T11:25:12.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.839+0000 7f862b4ad700 1 -- start start 2024-01-15T11:25:12.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f862b4ad700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f862406ee50 0x7f862415a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f862b4ad700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8624073f90 0x7f862415ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f862b4ad700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f862415c140 0x7f8624153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f862b4ad700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8624152200 con 0x7f862406ee50 2024-01-15T11:25:12.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f862b4ad700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86241534f0 con 0x7f8624073f90 2024-01-15T11:25:12.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f862b4ad700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8624152450 con 0x7f862415c140 2024-01-15T11:25:12.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8629249700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f862406ee50 0x7f862415a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8629249700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f862406ee50 0x7f862415a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36370/0 (socket says 172.21.15.113:36370) 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8629249700 1 -- 172.21.15.113:0/1625547008 learned_addr learned my addr 172.21.15.113:0/1625547008 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8628a48700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8624073f90 0x7f862415ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8629a4a700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f862415c140 0x7f8624153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8629249700 1 -- 172.21.15.113:0/1625547008 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f862415c140 msgr2=0x7f8624153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8629249700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f862415c140 0x7f8624153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8629249700 1 -- 172.21.15.113:0/1625547008 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8624073f90 msgr2=0x7f862415ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8629249700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8624073f90 0x7f862415ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8629249700 1 -- 172.21.15.113:0/1625547008 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8620009550 con 0x7f862406ee50 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8629249700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f862406ee50 0x7f862415a710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8620009130 tx=0x7f8620004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8619ffb700 1 -- 172.21.15.113:0/1625547008 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620011630 con 0x7f862406ee50 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8619ffb700 1 -- 172.21.15.113:0/1625547008 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8620011790 con 0x7f862406ee50 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f862b4ad700 1 -- 172.21.15.113:0/1625547008 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8624152730 con 0x7f862406ee50 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.840+0000 7f8619ffb700 1 -- 172.21.15.113:0/1625547008 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620011b20 con 0x7f862406ee50 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.841+0000 7f862b4ad700 1 -- 172.21.15.113:0/1625547008 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8624152cf0 con 0x7f862406ee50 2024-01-15T11:25:12.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.841+0000 7f862b4ad700 1 -- 172.21.15.113:0/1625547008 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8624067f60 con 0x7f862406ee50 2024-01-15T11:25:12.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.842+0000 7f8619ffb700 1 -- 172.21.15.113:0/1625547008 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8620021070 con 0x7f862406ee50 2024-01-15T11:25:12.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.842+0000 7f8619ffb700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f860806b350 0x7f860806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:12.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.842+0000 7f8619ffb700 1 -- 172.21.15.113:0/1625547008 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f862008b130 con 0x7f862406ee50 2024-01-15T11:25:12.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.842+0000 7f8628a48700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f860806b350 0x7f860806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:12.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.843+0000 7f8628a48700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f860806b350 0x7f860806d810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f862415f520 tx=0x7f8614009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:12.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:12.847+0000 7f8619ffb700 1 -- 172.21.15.113:0/1625547008 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8620056380 con 0x7f862406ee50 2024-01-15T11:25:13.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.040+0000 7f862b4ad700 1 -- 172.21.15.113:0/1625547008 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f86240d0860 con 0x7f862406ee50 2024-01-15T11:25:13.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.040+0000 7f8619ffb700 1 -- 172.21.15.113:0/1625547008 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f86200599a0 con 0x7f862406ee50 2024-01-15T11:25:13.039 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:13.039 INFO:teuthology.orchestra.run.smithi113.stdout:[ 2024-01-15T11:25:13.039 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:25:13.040 INFO:teuthology.orchestra.run.smithi113.stdout: "poolnum": 1, 2024-01-15T11:25:13.040 INFO:teuthology.orchestra.run.smithi113.stdout: "poolname": ".mgr" 2024-01-15T11:25:13.040 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:25:13.040 INFO:teuthology.orchestra.run.smithi113.stdout:] 2024-01-15T11:25:13.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.043+0000 7f862b4ad700 1 -- 172.21.15.113:0/1625547008 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f860806b350 msgr2=0x7f860806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:13.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.043+0000 7f862b4ad700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f860806b350 0x7f860806d810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f862415f520 tx=0x7f8614009040 comp rx=0 tx=0).stop 2024-01-15T11:25:13.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.043+0000 7f862b4ad700 1 -- 172.21.15.113:0/1625547008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f862406ee50 msgr2=0x7f862415a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:13.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.043+0000 7f862b4ad700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f862406ee50 0x7f862415a710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8620009130 tx=0x7f8620004750 comp rx=0 tx=0).stop 2024-01-15T11:25:13.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.043+0000 7f862b4ad700 1 -- 172.21.15.113:0/1625547008 shutdown_connections 2024-01-15T11:25:13.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.043+0000 7f862b4ad700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f860806b350 0x7f860806d810 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:13.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.043+0000 7f862b4ad700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f862415c140 0x7f8624153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:13.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.043+0000 7f862b4ad700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f862406ee50 0x7f862415a710 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:13.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.043+0000 7f862b4ad700 1 --2- 172.21.15.113:0/1625547008 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8624073f90 0x7f862415ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:13.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.043+0000 7f862b4ad700 1 -- 172.21.15.113:0/1625547008 >> 172.21.15.113:0/1625547008 conn(0x7f86240cd570 msgr2=0x7f86240cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:13.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.043+0000 7f862b4ad700 1 -- 172.21.15.113:0/1625547008 shutdown_connections 2024-01-15T11:25:13.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.043+0000 7f862b4ad700 1 -- 172.21.15.113:0/1625547008 wait complete. 2024-01-15T11:25:13.053 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T11:25:13.053 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T11:25:13.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.179+0000 7f05b713a700 1 Processor -- start 2024-01-15T11:25:13.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.179+0000 7f05b713a700 1 -- start start 2024-01-15T11:25:13.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05b713a700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f05b015bd70 0x7f05b006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:13.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05b713a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f05b00d07c0 0x7f05b00d1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05b713a700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f05b00d20e0 0x7f05b015f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05b713a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b00d2530 con 0x7f05b00d07c0 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05b713a700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b00aec00 con 0x7f05b00d20e0 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05b713a700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b00705d0 con 0x7f05b015bd70 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05affff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f05b00d07c0 0x7f05b00d1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05affff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f05b00d07c0 0x7f05b00d1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36386/0 (socket says 172.21.15.113:36386) 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05affff700 1 -- 172.21.15.113:0/2656963493 learned_addr learned my addr 172.21.15.113:0/2656963493 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05b56d7700 1 --2- 172.21.15.113:0/2656963493 >> v2:172.21.15.141:3300/0 conn(0x7f05b00d20e0 0x7f05b015f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05b4ed6700 1 --2- 172.21.15.113:0/2656963493 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f05b015bd70 0x7f05b006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05affff700 1 -- 172.21.15.113:0/2656963493 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f05b015bd70 msgr2=0x7f05b006fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05affff700 1 --2- 172.21.15.113:0/2656963493 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f05b015bd70 0x7f05b006fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05affff700 1 -- 172.21.15.113:0/2656963493 >> v2:172.21.15.141:3300/0 conn(0x7f05b00d20e0 msgr2=0x7f05b015f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05affff700 1 --2- 172.21.15.113:0/2656963493 >> v2:172.21.15.141:3300/0 conn(0x7f05b00d20e0 0x7f05b015f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05affff700 1 -- 172.21.15.113:0/2656963493 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b00708b0 con 0x7f05b00d07c0 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05affff700 1 --2- 172.21.15.113:0/2656963493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05b00d07c0 0x7f05b00d1ba0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f05b0072de0 tx=0x7f059c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d9a542e47cc123 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05af7fe700 1 -- 172.21.15.113:0/2656963493 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c00b870 con 0x7f05b00d07c0 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05af7fe700 1 -- 172.21.15.113:0/2656963493 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059c00b9d0 con 0x7f05b00d07c0 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.180+0000 7f05af7fe700 1 -- 172.21.15.113:0/2656963493 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c00bd60 con 0x7f05b00d07c0 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.181+0000 7f05b713a700 1 -- 172.21.15.113:0/2656963493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05b00d07c0 msgr2=0x7f05b00d1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.181+0000 7f05b713a700 1 --2- 172.21.15.113:0/2656963493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05b00d07c0 0x7f05b00d1ba0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f05b0072de0 tx=0x7f059c009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.181+0000 7f05b713a700 1 -- 172.21.15.113:0/2656963493 shutdown_connections 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.181+0000 7f05b713a700 1 --2- 172.21.15.113:0/2656963493 >> v2:172.21.15.141:3300/0 conn(0x7f05b00d20e0 0x7f05b015f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.181+0000 7f05b713a700 1 --2- 172.21.15.113:0/2656963493 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f05b015bd70 0x7f05b006fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.181+0000 7f05b713a700 1 --2- 172.21.15.113:0/2656963493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05b00d07c0 0x7f05b00d1ba0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.181+0000 7f05b713a700 1 -- 172.21.15.113:0/2656963493 >> 172.21.15.113:0/2656963493 conn(0x7f05b006dc80 msgr2=0x7f05b009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.181+0000 7f05b713a700 1 -- 172.21.15.113:0/2656963493 shutdown_connections 2024-01-15T11:25:13.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.181+0000 7f05b713a700 1 -- 172.21.15.113:0/2656963493 wait complete. 2024-01-15T11:25:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.182+0000 7f05b713a700 1 Processor -- start 2024-01-15T11:25:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.182+0000 7f05b713a700 1 -- start start 2024-01-15T11:25:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b713a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05b00d07c0 0x7f05b00b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b713a700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f05b00d20e0 0x7f05b00b7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b713a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05b015bd70 0x7f05b0150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b713a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b0159a10 con 0x7f05b00d07c0 2024-01-15T11:25:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b713a700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b00b6360 con 0x7f05b015bd70 2024-01-15T11:25:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b713a700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b00b5160 con 0x7f05b00d20e0 2024-01-15T11:25:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b4ed6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05b00d07c0 0x7f05b00b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b4ed6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05b00d07c0 0x7f05b00b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36400/0 (socket says 172.21.15.113:36400) 2024-01-15T11:25:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b4ed6700 1 -- 172.21.15.113:0/1931594230 learned_addr learned my addr 172.21.15.113:0/1931594230 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05affff700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f05b00d20e0 0x7f05b00b7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b56d7700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05b015bd70 0x7f05b0150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b4ed6700 1 -- 172.21.15.113:0/1931594230 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f05b00d20e0 msgr2=0x7f05b00b7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b4ed6700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f05b00d20e0 0x7f05b00b7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b4ed6700 1 -- 172.21.15.113:0/1931594230 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05b015bd70 msgr2=0x7f05b0150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b4ed6700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05b015bd70 0x7f05b0150dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b4ed6700 1 -- 172.21.15.113:0/1931594230 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c009550 con 0x7f05b00d07c0 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.183+0000 7f05b4ed6700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05b00d07c0 0x7f05b00b4b80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05a400c7f0 tx=0x7f05a400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.184+0000 7f05ad7fa700 1 -- 172.21.15.113:0/1931594230 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4011800 con 0x7f05b00d07c0 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.184+0000 7f05ad7fa700 1 -- 172.21.15.113:0/1931594230 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a4011960 con 0x7f05b00d07c0 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.184+0000 7f05b713a700 1 -- 172.21.15.113:0/1931594230 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b0159070 con 0x7f05b00d07c0 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.184+0000 7f05ad7fa700 1 -- 172.21.15.113:0/1931594230 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4011cf0 con 0x7f05b00d07c0 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.184+0000 7f05b713a700 1 -- 172.21.15.113:0/1931594230 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b0159630 con 0x7f05b00d07c0 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.185+0000 7f05b713a700 1 -- 172.21.15.113:0/1931594230 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b0067f60 con 0x7f05b00d07c0 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.185+0000 7f05ad7fa700 1 -- 172.21.15.113:0/1931594230 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f05a4018360 con 0x7f05b00d07c0 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.185+0000 7f05ad7fa700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f059406b350 0x7f059406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.185+0000 7f05ad7fa700 1 -- 172.21.15.113:0/1931594230 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f05a4089640 con 0x7f05b00d07c0 2024-01-15T11:25:13.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.185+0000 7f05affff700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f059406b350 0x7f059406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:13.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.186+0000 7f05affff700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f059406b350 0x7f059406d810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f059c000c00 tx=0x7f059c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:13.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.190+0000 7f05ad7fa700 1 -- 172.21.15.113:0/1931594230 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05a4054890 con 0x7f05b00d07c0 2024-01-15T11:25:13.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:13.380+0000 7f05b713a700 1 -- 172.21.15.113:0/1931594230 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f05b015d3e0 con 0x7f05b00d07c0 2024-01-15T11:25:14.022 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:25:14.025 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:25:14.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.210+0000 7f05ad7fa700 1 -- 172.21.15.113:0/1931594230 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f05a4057eb0 con 0x7f05b00d07c0 2024-01-15T11:25:14.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.213+0000 7f05b713a700 1 -- 172.21.15.113:0/1931594230 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f059406b350 msgr2=0x7f059406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:14.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.213+0000 7f05b713a700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f059406b350 0x7f059406d810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f059c000c00 tx=0x7f059c008030 comp rx=0 tx=0).stop 2024-01-15T11:25:14.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.213+0000 7f05b713a700 1 -- 172.21.15.113:0/1931594230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05b00d07c0 msgr2=0x7f05b00b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:14.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.213+0000 7f05b713a700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05b00d07c0 0x7f05b00b4b80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05a400c7f0 tx=0x7f05a400cbb0 comp rx=0 tx=0).stop 2024-01-15T11:25:14.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.213+0000 7f05b713a700 1 -- 172.21.15.113:0/1931594230 shutdown_connections 2024-01-15T11:25:14.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.213+0000 7f05b713a700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f059406b350 0x7f059406d810 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f059c000c00 tx=0x7f059c008030 comp rx=0 tx=0).stop 2024-01-15T11:25:14.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.213+0000 7f05b713a700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05b015bd70 0x7f05b0150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:14.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.213+0000 7f05b713a700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05b00d07c0 0x7f05b00b4b80 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05a400c7f0 tx=0x7f05a400cbb0 comp rx=0 tx=0).stop 2024-01-15T11:25:14.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.213+0000 7f05b713a700 1 --2- 172.21.15.113:0/1931594230 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f05b00d20e0 0x7f05b00b7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:14.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.213+0000 7f05b713a700 1 -- 172.21.15.113:0/1931594230 >> 172.21.15.113:0/1931594230 conn(0x7f05b006dc80 msgr2=0x7f05b006f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:14.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.214+0000 7f05b713a700 1 -- 172.21.15.113:0/1931594230 shutdown_connections 2024-01-15T11:25:14.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.215+0000 7f05b713a700 1 -- 172.21.15.113:0/1931594230 wait complete. 2024-01-15T11:25:14.220 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'cephfs_metadata' created 2024-01-15T11:25:14.231 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T11:25:14.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.365+0000 7f92f36c3700 1 Processor -- start 2024-01-15T11:25:14.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.365+0000 7f92f36c3700 1 -- start start 2024-01-15T11:25:14.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.366+0000 7f92f36c3700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f92ec0afb30 0x7f92ec073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:14.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.366+0000 7f92f36c3700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ec06fb30 con 0x7f92ec15e110 2024-01-15T11:25:14.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.366+0000 7f92f36c3700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ec06d980 con 0x7f92ec06f680 2024-01-15T11:25:14.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.366+0000 7f92f36c3700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92ec0b0260 con 0x7f92ec0afb30 2024-01-15T11:25:14.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.366+0000 7f92f145f700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f92ec15e110 0x7f92ec06aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:37430/0 (socket says 172.21.15.113:37430) 2024-01-15T11:25:14.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.366+0000 7f92f145f700 1 -- 172.21.15.113:0/2275756113 learned_addr learned my addr 172.21.15.113:0/2275756113 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:14.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.366+0000 7f92f1c60700 1 --2- 172.21.15.113:0/2275756113 >> v2:172.21.15.113:3300/0 conn(0x7f92ec0afb30 0x7f92ec073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:14.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.367+0000 7f92e3fff700 1 -- 172.21.15.113:0/2275756113 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068821142 0 0) 0x7f92ec06fb30 con 0x7f92ec15e110 2024-01-15T11:25:14.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.367+0000 7f92e3fff700 1 -- 172.21.15.113:0/2275756113 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d80035b0 con 0x7f92ec15e110 2024-01-15T11:25:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.367+0000 7f92f1c60700 1 -- 172.21.15.113:0/2275756113 >> v1:172.21.15.141:6789/0 conn(0x7f92ec06f680 legacy=0x7f92ec06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.367+0000 7f92f1c60700 1 -- 172.21.15.113:0/2275756113 >> v1:172.21.15.113:6789/0 conn(0x7f92ec15e110 legacy=0x7f92ec06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.367+0000 7f92f1c60700 1 -- 172.21.15.113:0/2275756113 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ec0b0540 con 0x7f92ec0afb30 2024-01-15T11:25:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.367+0000 7f92f1c60700 1 --2- 172.21.15.113:0/2275756113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0afb30 0x7f92ec073f90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f92e4009d90 tx=0x7f92e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb0fbb0eff5bbe4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.367+0000 7f92e3fff700 1 -- 172.21.15.113:0/2275756113 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e400e070 con 0x7f92ec0afb30 2024-01-15T11:25:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.367+0000 7f92e3fff700 1 -- 172.21.15.113:0/2275756113 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e400bb20 con 0x7f92ec0afb30 2024-01-15T11:25:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.367+0000 7f92e3fff700 1 -- 172.21.15.113:0/2275756113 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e4012430 con 0x7f92ec0afb30 2024-01-15T11:25:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.368+0000 7f92f36c3700 1 -- 172.21.15.113:0/2275756113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0afb30 msgr2=0x7f92ec073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.368+0000 7f92f36c3700 1 --2- 172.21.15.113:0/2275756113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0afb30 0x7f92ec073f90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f92e4009d90 tx=0x7f92e400b330 comp rx=0 tx=0).stop 2024-01-15T11:25:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.368+0000 7f92f36c3700 1 -- 172.21.15.113:0/2275756113 shutdown_connections 2024-01-15T11:25:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.368+0000 7f92f36c3700 1 --2- 172.21.15.113:0/2275756113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0afb30 0x7f92ec073f90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.368+0000 7f92f36c3700 1 -- 172.21.15.113:0/2275756113 >> 172.21.15.113:0/2275756113 conn(0x7f92ec0b3250 msgr2=0x7f92ec0d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:14.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.368+0000 7f92f36c3700 1 -- 172.21.15.113:0/2275756113 shutdown_connections 2024-01-15T11:25:14.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.368+0000 7f92f36c3700 1 -- 172.21.15.113:0/2275756113 wait complete. 2024-01-15T11:25:14.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.369+0000 7f92f36c3700 1 Processor -- start 2024-01-15T11:25:14.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.369+0000 7f92f36c3700 1 -- start start 2024-01-15T11:25:14.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f36c3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec06f680 0x7f92ec1533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:14.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f36c3700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f92ec0afb30 0x7f92ec1f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:14.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f36c3700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92ec15e110 0x7f92ec0b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:14.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f36c3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec1597b0 con 0x7f92ec06f680 2024-01-15T11:25:14.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f36c3700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec1538f0 con 0x7f92ec15e110 2024-01-15T11:25:14.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f36c3700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec153c70 con 0x7f92ec0afb30 2024-01-15T11:25:14.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f145f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec06f680 0x7f92ec1533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:14.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f145f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec06f680 0x7f92ec1533b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45356/0 (socket says 172.21.15.113:45356) 2024-01-15T11:25:14.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f145f700 1 -- 172.21.15.113:0/1002981199 learned_addr learned my addr 172.21.15.113:0/1002981199 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f145f700 1 -- 172.21.15.113:0/1002981199 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f92ec0afb30 msgr2=0x7f92ec1f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f145f700 1 --2- 172.21.15.113:0/1002981199 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f92ec0afb30 0x7f92ec1f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f145f700 1 -- 172.21.15.113:0/1002981199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92ec15e110 msgr2=0x7f92ec0b1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f145f700 1 --2- 172.21.15.113:0/1002981199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92ec15e110 0x7f92ec0b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f145f700 1 -- 172.21.15.113:0/1002981199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e4009a70 con 0x7f92ec06f680 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f145f700 1 --2- 172.21.15.113:0/1002981199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec06f680 0x7f92ec1533b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f92dc00c700 tx=0x7f92dc00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92e1ffb700 1 -- 172.21.15.113:0/1002981199 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92dc005d70 con 0x7f92ec06f680 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92e1ffb700 1 -- 172.21.15.113:0/1002981199 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92dc0070d0 con 0x7f92ec06f680 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f36c3700 1 -- 172.21.15.113:0/1002981199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ec1590f0 con 0x7f92ec06f680 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92e1ffb700 1 -- 172.21.15.113:0/1002981199 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92dc007460 con 0x7f92ec06f680 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.370+0000 7f92f36c3700 1 -- 172.21.15.113:0/1002981199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ec15b0d0 con 0x7f92ec06f680 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.371+0000 7f92f36c3700 1 -- 172.21.15.113:0/1002981199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ec1615a0 con 0x7f92ec06f680 2024-01-15T11:25:14.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.372+0000 7f92e1ffb700 1 -- 172.21.15.113:0/1002981199 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f92dc0075c0 con 0x7f92ec06f680 2024-01-15T11:25:14.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.372+0000 7f92e1ffb700 1 --2- 172.21.15.113:0/1002981199 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f92d006b350 0x7f92d006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:14.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.372+0000 7f92e1ffb700 1 -- 172.21.15.113:0/1002981199 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f92dc089070 con 0x7f92ec06f680 2024-01-15T11:25:14.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.372+0000 7f92f0c5e700 1 --2- 172.21.15.113:0/1002981199 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f92d006b350 0x7f92d006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:14.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.373+0000 7f92f0c5e700 1 --2- 172.21.15.113:0/1002981199 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f92d006b350 0x7f92d006d810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f92e8000f60 tx=0x7f92e8007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:14.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.377+0000 7f92e1ffb700 1 -- 172.21.15.113:0/1002981199 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92dc0bbe90 con 0x7f92ec06f680 2024-01-15T11:25:14.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:14.571+0000 7f92f36c3700 1 -- 172.21.15.113:0/1002981199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f92ec068670 con 0x7f92ec06f680 2024-01-15T11:25:15.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.212+0000 7f92e1ffb700 1 -- 172.21.15.113:0/1002981199 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f92dc054030 con 0x7f92ec06f680 2024-01-15T11:25:15.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.215+0000 7f92f36c3700 1 -- 172.21.15.113:0/1002981199 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f92d006b350 msgr2=0x7f92d006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:15.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.215+0000 7f92f36c3700 1 --2- 172.21.15.113:0/1002981199 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f92d006b350 0x7f92d006d810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f92e8000f60 tx=0x7f92e8007a40 comp rx=0 tx=0).stop 2024-01-15T11:25:15.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.215+0000 7f92f36c3700 1 -- 172.21.15.113:0/1002981199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec06f680 msgr2=0x7f92ec1533b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:15.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.215+0000 7f92f36c3700 1 --2- 172.21.15.113:0/1002981199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec06f680 0x7f92ec1533b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f92dc00c700 tx=0x7f92dc00cac0 comp rx=0 tx=0).stop 2024-01-15T11:25:15.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.215+0000 7f92f36c3700 1 -- 172.21.15.113:0/1002981199 shutdown_connections 2024-01-15T11:25:15.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.215+0000 7f92f36c3700 1 --2- 172.21.15.113:0/1002981199 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f92d006b350 0x7f92d006d810 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f92e8000f60 tx=0x7f92e8007a40 comp rx=0 tx=0).stop 2024-01-15T11:25:15.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.215+0000 7f92f36c3700 1 --2- 172.21.15.113:0/1002981199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92ec15e110 0x7f92ec0b1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:15.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.215+0000 7f92f36c3700 1 --2- 172.21.15.113:0/1002981199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec06f680 0x7f92ec1533b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:15.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.215+0000 7f92f36c3700 1 --2- 172.21.15.113:0/1002981199 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f92ec0afb30 0x7f92ec1f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:15.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.215+0000 7f92f36c3700 1 -- 172.21.15.113:0/1002981199 >> 172.21.15.113:0/1002981199 conn(0x7f92ec0b3250 msgr2=0x7f92ec073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:15.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.216+0000 7f92f36c3700 1 -- 172.21.15.113:0/1002981199 shutdown_connections 2024-01-15T11:25:15.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.217+0000 7f92f36c3700 1 -- 172.21.15.113:0/1002981199 wait complete. 2024-01-15T11:25:15.218 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'cephfs_data' created 2024-01-15T11:25:15.238 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T11:25:15.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.364+0000 7f3f61108700 1 Processor -- start 2024-01-15T11:25:15.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.364+0000 7f3f61108700 1 -- start start 2024-01-15T11:25:15.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.365+0000 7f3f61108700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3f5c161510 0x7f3f5c0cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:15.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.365+0000 7f3f61108700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3f5c0b98e0 0x7f3f5c0cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:15.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.365+0000 7f3f61108700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3f5c0b59d0 0x7f3f5c0b5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:15.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.365+0000 7f3f61108700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5c162af0 con 0x7f3f5c0b98e0 2024-01-15T11:25:15.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.365+0000 7f3f61108700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5c0d13e0 con 0x7f3f5c161510 2024-01-15T11:25:15.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.365+0000 7f3f61108700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c161f70 con 0x7f3f5c0b59d0 2024-01-15T11:25:15.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.365+0000 7f3f5a59c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3f5c0b98e0 0x7f3f5c0cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:15.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.365+0000 7f3f5a59c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3f5c0b98e0 0x7f3f5c0cf160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45366/0 (socket says 172.21.15.113:45366) 2024-01-15T11:25:15.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.365+0000 7f3f5a59c700 1 -- 172.21.15.113:0/2506224087 learned_addr learned my addr 172.21.15.113:0/2506224087 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:15.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f5ad9d700 1 --2- 172.21.15.113:0/2506224087 >> v2:172.21.15.141:3300/0 conn(0x7f3f5c161510 0x7f3f5c0cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:15.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f5b59e700 1 --2- 172.21.15.113:0/2506224087 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3f5c0b59d0 0x7f3f5c0b5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f5a59c700 1 -- 172.21.15.113:0/2506224087 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3f5c0b59d0 msgr2=0x7f3f5c0b5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f5a59c700 1 --2- 172.21.15.113:0/2506224087 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3f5c0b59d0 0x7f3f5c0b5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f5a59c700 1 -- 172.21.15.113:0/2506224087 >> v2:172.21.15.141:3300/0 conn(0x7f3f5c161510 msgr2=0x7f3f5c0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f5a59c700 1 --2- 172.21.15.113:0/2506224087 >> v2:172.21.15.141:3300/0 conn(0x7f3f5c161510 0x7f3f5c0cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f5a59c700 1 -- 172.21.15.113:0/2506224087 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c162250 con 0x7f3f5c0b98e0 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f5ad9d700 1 --2- 172.21.15.113:0/2506224087 >> v2:172.21.15.141:3300/0 conn(0x7f3f5c161510 0x7f3f5c0cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f5a59c700 1 --2- 172.21.15.113:0/2506224087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c0b98e0 0x7f3f5c0cf160 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c0d49f0 tx=0x7f3f44009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce64642b884a9b45 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f59d9b700 1 -- 172.21.15.113:0/2506224087 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4400b870 con 0x7f3f5c0b98e0 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f59d9b700 1 -- 172.21.15.113:0/2506224087 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4400b9d0 con 0x7f3f5c0b98e0 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f59d9b700 1 -- 172.21.15.113:0/2506224087 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4400bd60 con 0x7f3f5c0b98e0 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f61108700 1 -- 172.21.15.113:0/2506224087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c0b98e0 msgr2=0x7f3f5c0cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f61108700 1 --2- 172.21.15.113:0/2506224087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c0b98e0 0x7f3f5c0cf160 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c0d49f0 tx=0x7f3f44009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f61108700 1 -- 172.21.15.113:0/2506224087 shutdown_connections 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f61108700 1 --2- 172.21.15.113:0/2506224087 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3f5c0b59d0 0x7f3f5c0b5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:15.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f61108700 1 --2- 172.21.15.113:0/2506224087 >> v2:172.21.15.141:3300/0 conn(0x7f3f5c161510 0x7f3f5c0cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:15.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f61108700 1 --2- 172.21.15.113:0/2506224087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c0b98e0 0x7f3f5c0cf160 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:15.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.366+0000 7f3f61108700 1 -- 172.21.15.113:0/2506224087 >> 172.21.15.113:0/2506224087 conn(0x7f3f5c06b410 msgr2=0x7f3f5c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:15.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.367+0000 7f3f61108700 1 -- 172.21.15.113:0/2506224087 shutdown_connections 2024-01-15T11:25:15.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.367+0000 7f3f61108700 1 -- 172.21.15.113:0/2506224087 wait complete. 2024-01-15T11:25:15.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.368+0000 7f3f61108700 1 Processor -- start 2024-01-15T11:25:15.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.368+0000 7f3f61108700 1 -- start start 2024-01-15T11:25:15.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f61108700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3f5c0b59d0 0x7f3f5c0c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:15.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f61108700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f5c0b98e0 0x7f3f5c15f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:15.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f61108700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c161510 0x7f3f5c156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:15.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f61108700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c0c8060 con 0x7f3f5c161510 2024-01-15T11:25:15.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f61108700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c0c8340 con 0x7f3f5c0b98e0 2024-01-15T11:25:15.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f61108700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c0c8620 con 0x7f3f5c0b59d0 2024-01-15T11:25:15.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f5b59e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c161510 0x7f3f5c156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:15.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f5b59e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c161510 0x7f3f5c156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45370/0 (socket says 172.21.15.113:45370) 2024-01-15T11:25:15.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f5b59e700 1 -- 172.21.15.113:0/3090077451 learned_addr learned my addr 172.21.15.113:0/3090077451 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:15.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f5a59c700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f5c0b98e0 0x7f3f5c15f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:15.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f5ad9d700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3f5c0b59d0 0x7f3f5c0c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f5b59e700 1 -- 172.21.15.113:0/3090077451 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3f5c0b59d0 msgr2=0x7f3f5c0c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f5b59e700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3f5c0b59d0 0x7f3f5c0c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f5b59e700 1 -- 172.21.15.113:0/3090077451 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f5c0b98e0 msgr2=0x7f3f5c15f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f5b59e700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f5c0b98e0 0x7f3f5c15f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f5b59e700 1 -- 172.21.15.113:0/3090077451 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f44009550 con 0x7f3f5c161510 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f5ad9d700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3f5c0b59d0 0x7f3f5c0c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f5b59e700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c161510 0x7f3f5c156980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400ccf0 tx=0x7f3f5400a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f537fe700 1 -- 172.21.15.113:0/3090077451 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400ae10 con 0x7f3f5c161510 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f537fe700 1 -- 172.21.15.113:0/3090077451 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f54003a20 con 0x7f3f5c161510 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f61108700 1 -- 172.21.15.113:0/3090077451 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5c0c8900 con 0x7f3f5c161510 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.369+0000 7f3f537fe700 1 -- 172.21.15.113:0/3090077451 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54003db0 con 0x7f3f5c161510 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.370+0000 7f3f61108700 1 -- 172.21.15.113:0/3090077451 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5c15fa70 con 0x7f3f5c161510 2024-01-15T11:25:15.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.370+0000 7f3f61108700 1 -- 172.21.15.113:0/3090077451 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5c0c91b0 con 0x7f3f5c161510 2024-01-15T11:25:15.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.371+0000 7f3f537fe700 1 -- 172.21.15.113:0/3090077451 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3f54003f10 con 0x7f3f5c161510 2024-01-15T11:25:15.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.372+0000 7f3f537fe700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3f3c06b350 0x7f3f3c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:15.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.372+0000 7f3f537fe700 1 -- 172.21.15.113:0/3090077451 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f3f54012070 con 0x7f3f5c161510 2024-01-15T11:25:15.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.372+0000 7f3f5ad9d700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3f3c06b350 0x7f3f3c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:15.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.372+0000 7f3f5ad9d700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3f3c06b350 0x7f3f3c06d810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00aa20 tx=0x7f3f4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:15.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.377+0000 7f3f537fe700 1 -- 172.21.15.113:0/3090077451 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f54055580 con 0x7f3f5c161510 2024-01-15T11:25:15.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:15.609+0000 7f3f61108700 1 -- 172.21.15.113:0/3090077451 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3f5c068450 con 0x7f3f5c161510 2024-01-15T11:25:16.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.249+0000 7f3f537fe700 1 -- 172.21.15.113:0/3090077451 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3f54058ba0 con 0x7f3f5c161510 2024-01-15T11:25:16.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.254+0000 7f3f61108700 1 -- 172.21.15.113:0/3090077451 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3f3c06b350 msgr2=0x7f3f3c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:16.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.254+0000 7f3f61108700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3f3c06b350 0x7f3f3c06d810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00aa20 tx=0x7f3f4c008040 comp rx=0 tx=0).stop 2024-01-15T11:25:16.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.254+0000 7f3f61108700 1 -- 172.21.15.113:0/3090077451 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c161510 msgr2=0x7f3f5c156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:16.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.254+0000 7f3f61108700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c161510 0x7f3f5c156980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400ccf0 tx=0x7f3f5400a290 comp rx=0 tx=0).stop 2024-01-15T11:25:16.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.254+0000 7f3f61108700 1 -- 172.21.15.113:0/3090077451 shutdown_connections 2024-01-15T11:25:16.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.254+0000 7f3f61108700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3f3c06b350 0x7f3f3c06d810 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.254+0000 7f3f61108700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c161510 0x7f3f5c156980 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.254+0000 7f3f61108700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3f5c0b59d0 0x7f3f5c0c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.254+0000 7f3f61108700 1 --2- 172.21.15.113:0/3090077451 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f5c0b98e0 0x7f3f5c15f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.254+0000 7f3f61108700 1 -- 172.21.15.113:0/3090077451 >> 172.21.15.113:0/3090077451 conn(0x7f3f5c06b410 msgr2=0x7f3f5c0b3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:16.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.255+0000 7f3f61108700 1 -- 172.21.15.113:0/3090077451 shutdown_connections 2024-01-15T11:25:16.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.256+0000 7f3f61108700 1 -- 172.21.15.113:0/3090077451 wait complete. 2024-01-15T11:25:16.257 INFO:teuthology.orchestra.run.smithi113.stderr:new fs with metadata pool 2 and data pool 3 2024-01-15T11:25:16.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:16.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.400+0000 7f6da4aef700 1 Processor -- start 2024-01-15T11:25:16.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.400+0000 7f6da4aef700 1 -- start start 2024-01-15T11:25:16.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6da4aef700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6da01473a0 0x7f6da015e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6da4aef700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6da0147780 0x7f6da006c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6da4aef700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6da006dc20 0x7f6da0073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6da4aef700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6da006c9a0 con 0x7f6da0147780 2024-01-15T11:25:16.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6da4aef700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6da015eb00 con 0x7f6da01473a0 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6da4aef700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da015ede0 con 0x7f6da006dc20 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6d9dd9b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6da0147780 0x7f6da006c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6d9dd9b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6da0147780 0x7f6da006c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45382/0 (socket says 172.21.15.113:45382) 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6d9dd9b700 1 -- 172.21.15.113:0/1282322896 learned_addr learned my addr 172.21.15.113:0/1282322896 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6d9ed9d700 1 --2- 172.21.15.113:0/1282322896 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6da006dc20 0x7f6da0073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6d9e59c700 1 --2- 172.21.15.113:0/1282322896 >> v2:172.21.15.141:3300/0 conn(0x7f6da01473a0 0x7f6da015e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6d9dd9b700 1 -- 172.21.15.113:0/1282322896 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6da006dc20 msgr2=0x7f6da0073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6d9dd9b700 1 --2- 172.21.15.113:0/1282322896 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6da006dc20 0x7f6da0073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6d9dd9b700 1 -- 172.21.15.113:0/1282322896 >> v2:172.21.15.141:3300/0 conn(0x7f6da01473a0 msgr2=0x7f6da015e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6d9dd9b700 1 --2- 172.21.15.113:0/1282322896 >> v2:172.21.15.141:3300/0 conn(0x7f6da01473a0 0x7f6da015e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6d9dd9b700 1 -- 172.21.15.113:0/1282322896 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da015f0c0 con 0x7f6da0147780 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6d9dd9b700 1 --2- 172.21.15.113:0/1282322896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6da0147780 0x7f6da006c460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6da0072f90 tx=0x7f6d94009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8613df1aecec68e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:16.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.401+0000 7f6d9d59a700 1 -- 172.21.15.113:0/1282322896 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9400b870 con 0x7f6da0147780 2024-01-15T11:25:16.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.402+0000 7f6d9d59a700 1 -- 172.21.15.113:0/1282322896 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9400b9d0 con 0x7f6da0147780 2024-01-15T11:25:16.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.402+0000 7f6d9d59a700 1 -- 172.21.15.113:0/1282322896 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9400bd60 con 0x7f6da0147780 2024-01-15T11:25:16.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.402+0000 7f6da4aef700 1 -- 172.21.15.113:0/1282322896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6da0147780 msgr2=0x7f6da006c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:16.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.402+0000 7f6da4aef700 1 --2- 172.21.15.113:0/1282322896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6da0147780 0x7f6da006c460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6da0072f90 tx=0x7f6d94009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.402+0000 7f6da4aef700 1 -- 172.21.15.113:0/1282322896 shutdown_connections 2024-01-15T11:25:16.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.402+0000 7f6da4aef700 1 --2- 172.21.15.113:0/1282322896 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6da006dc20 0x7f6da0073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.402+0000 7f6da4aef700 1 --2- 172.21.15.113:0/1282322896 >> v2:172.21.15.141:3300/0 conn(0x7f6da01473a0 0x7f6da015e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.402+0000 7f6da4aef700 1 --2- 172.21.15.113:0/1282322896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6da0147780 0x7f6da006c460 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.402+0000 7f6da4aef700 1 -- 172.21.15.113:0/1282322896 >> 172.21.15.113:0/1282322896 conn(0x7f6da00b3130 msgr2=0x7f6da00d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:16.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.402+0000 7f6da4aef700 1 -- 172.21.15.113:0/1282322896 shutdown_connections 2024-01-15T11:25:16.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.402+0000 7f6da4aef700 1 -- 172.21.15.113:0/1282322896 wait complete. 2024-01-15T11:25:16.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.403+0000 7f6da4aef700 1 Processor -- start 2024-01-15T11:25:16.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.403+0000 7f6da4aef700 1 -- start start 2024-01-15T11:25:16.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.404+0000 7f6da4aef700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6da006dc20 0x7f6da015d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.404+0000 7f6da4aef700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6da01473a0 0x7f6da00ae020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.404+0000 7f6da4aef700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6da0147780 0x7f6da0160ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.404+0000 7f6da4aef700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da00af5a0 con 0x7f6da006dc20 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.404+0000 7f6da4aef700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da015dce0 con 0x7f6da0147780 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.404+0000 7f6da4aef700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da00ae920 con 0x7f6da01473a0 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.404+0000 7f6d9e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6da006dc20 0x7f6da015d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.404+0000 7f6d9e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6da006dc20 0x7f6da015d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45396/0 (socket says 172.21.15.113:45396) 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.404+0000 7f6d9e59c700 1 -- 172.21.15.113:0/3700827554 learned_addr learned my addr 172.21.15.113:0/3700827554 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6d9ed9d700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6da0147780 0x7f6da0160ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6d9dd9b700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6da01473a0 0x7f6da00ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6d9e59c700 1 -- 172.21.15.113:0/3700827554 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6da01473a0 msgr2=0x7f6da00ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6d9e59c700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6da01473a0 0x7f6da00ae020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6d9e59c700 1 -- 172.21.15.113:0/3700827554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6da0147780 msgr2=0x7f6da0160ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:16.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6d9e59c700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6da0147780 0x7f6da0160ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6d9e59c700 1 -- 172.21.15.113:0/3700827554 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d94009550 con 0x7f6da006dc20 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6d9e59c700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6da006dc20 0x7f6da015d7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d9000e810 tx=0x7f6d9000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6d8effd700 1 -- 172.21.15.113:0/3700827554 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d90013610 con 0x7f6da006dc20 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6d8effd700 1 -- 172.21.15.113:0/3700827554 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d90013770 con 0x7f6da006dc20 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6da4aef700 1 -- 172.21.15.113:0/3700827554 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da00aec00 con 0x7f6da006dc20 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6d8effd700 1 -- 172.21.15.113:0/3700827554 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d90013b20 con 0x7f6da006dc20 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.405+0000 7f6da4aef700 1 -- 172.21.15.113:0/3700827554 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da00af1c0 con 0x7f6da006dc20 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.406+0000 7f6da4aef700 1 -- 172.21.15.113:0/3700827554 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da0067f60 con 0x7f6da006dc20 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.406+0000 7f6d8effd700 1 -- 172.21.15.113:0/3700827554 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6d90013c80 con 0x7f6da006dc20 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.407+0000 7f6d8effd700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6d8006b350 0x7f6d8006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.407+0000 7f6d8effd700 1 -- 172.21.15.113:0/3700827554 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d900897e0 con 0x7f6da006dc20 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.407+0000 7f6d9dd9b700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6d8006b350 0x7f6d8006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.407+0000 7f6d9dd9b700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6d8006b350 0x7f6d8006d810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6da0072f90 tx=0x7f6d94008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:16.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.411+0000 7f6d8effd700 1 -- 172.21.15.113:0/3700827554 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d90054660 con 0x7f6da006dc20 2024-01-15T11:25:16.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.604+0000 7f6da4aef700 1 -- 172.21.15.113:0/3700827554 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6da00af810 con 0x7f6da006dc20 2024-01-15T11:25:16.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.605+0000 7f6d8effd700 1 -- 172.21.15.113:0/3700827554 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16578 (secure 0 0 0) 0x7f6d90057c80 con 0x7f6da006dc20 2024-01-15T11:25:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:16.603 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":19,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:15.611278+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:16.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.607+0000 7f6da4aef700 1 -- 172.21.15.113:0/3700827554 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6d8006b350 msgr2=0x7f6d8006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:16.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.607+0000 7f6da4aef700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6d8006b350 0x7f6d8006d810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6da0072f90 tx=0x7f6d94008030 comp rx=0 tx=0).stop 2024-01-15T11:25:16.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.607+0000 7f6da4aef700 1 -- 172.21.15.113:0/3700827554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6da006dc20 msgr2=0x7f6da015d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:16.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.607+0000 7f6da4aef700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6da006dc20 0x7f6da015d7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d9000e810 tx=0x7f6d9000ebd0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.607+0000 7f6da4aef700 1 -- 172.21.15.113:0/3700827554 shutdown_connections 2024-01-15T11:25:16.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.607+0000 7f6da4aef700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6d8006b350 0x7f6d8006d810 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.607+0000 7f6da4aef700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6da0147780 0x7f6da0160ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.607+0000 7f6da4aef700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6da006dc20 0x7f6da015d7a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.607+0000 7f6da4aef700 1 --2- 172.21.15.113:0/3700827554 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6da01473a0 0x7f6da00ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.607+0000 7f6da4aef700 1 -- 172.21.15.113:0/3700827554 >> 172.21.15.113:0/3700827554 conn(0x7f6da00b3130 msgr2=0x7f6da015f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:16.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.607+0000 7f6da4aef700 1 -- 172.21.15.113:0/3700827554 shutdown_connections 2024-01-15T11:25:16.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.607+0000 7f6da4aef700 1 -- 172.21.15.113:0/3700827554 wait complete. 2024-01-15T11:25:16.617 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:16.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.740+0000 7f233f0e6700 1 Processor -- start 2024-01-15T11:25:16.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.740+0000 7f233f0e6700 1 -- start start 2024-01-15T11:25:16.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.740+0000 7f233f0e6700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f23380ce090 0x7f233806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.740+0000 7f233f0e6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2338073f90 0x7f2338074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.740+0000 7f233f0e6700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233806e7f0 con 0x7f233815e2a0 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.740+0000 7f233f0e6700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2338074960 con 0x7f2338073f90 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.740+0000 7f233f0e6700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f233806dec0 con 0x7f23380ce090 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.741+0000 7f233d683700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2338073f90 0x7f2338074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.741+0000 7f233d683700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2338073f90 0x7f2338074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45404/0 (socket says 172.21.15.113:45404) 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.741+0000 7f233d683700 1 -- 172.21.15.113:0/2454911307 learned_addr learned my addr 172.21.15.113:0/2454911307 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.741+0000 7f2337fff700 1 --2- 172.21.15.113:0/2454911307 >> v2:172.21.15.141:3300/0 conn(0x7f23380ce090 0x7f233806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.741+0000 7f233d683700 1 -- 172.21.15.113:0/2454911307 >> v2:172.21.15.141:3300/0 conn(0x7f23380ce090 msgr2=0x7f233806ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.741+0000 7f233d683700 1 --2- 172.21.15.113:0/2454911307 >> v2:172.21.15.141:3300/0 conn(0x7f23380ce090 0x7f233806ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.741+0000 7f233d683700 1 -- 172.21.15.113:0/2454911307 >> v1:172.21.15.141:6789/0 conn(0x7f233815e2a0 legacy=0x7f233806ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.741+0000 7f233d683700 1 -- 172.21.15.113:0/2454911307 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233806e1a0 con 0x7f2338073f90 2024-01-15T11:25:16.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.741+0000 7f233d683700 1 --2- 172.21.15.113:0/2454911307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2338073f90 0x7f2338074420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2338071840 tx=0x7f233000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72160af941fe95e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:16.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.741+0000 7f23377fe700 1 -- 172.21.15.113:0/2454911307 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233000e070 con 0x7f2338073f90 2024-01-15T11:25:16.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.741+0000 7f23377fe700 1 -- 172.21.15.113:0/2454911307 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2330003a20 con 0x7f2338073f90 2024-01-15T11:25:16.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.741+0000 7f23377fe700 1 -- 172.21.15.113:0/2454911307 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330003dd0 con 0x7f2338073f90 2024-01-15T11:25:16.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.742+0000 7f233f0e6700 1 -- 172.21.15.113:0/2454911307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2338073f90 msgr2=0x7f2338074420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:16.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.742+0000 7f233f0e6700 1 --2- 172.21.15.113:0/2454911307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2338073f90 0x7f2338074420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2338071840 tx=0x7f233000b330 comp rx=0 tx=0).stop 2024-01-15T11:25:16.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.742+0000 7f233f0e6700 1 -- 172.21.15.113:0/2454911307 shutdown_connections 2024-01-15T11:25:16.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.742+0000 7f233f0e6700 1 --2- 172.21.15.113:0/2454911307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2338073f90 0x7f2338074420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.742+0000 7f233f0e6700 1 --2- 172.21.15.113:0/2454911307 >> v2:172.21.15.141:3300/0 conn(0x7f23380ce090 0x7f233806ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.742+0000 7f233f0e6700 1 -- 172.21.15.113:0/2454911307 >> 172.21.15.113:0/2454911307 conn(0x7f23380cd570 msgr2=0x7f23380cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:16.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.742+0000 7f233f0e6700 1 -- 172.21.15.113:0/2454911307 shutdown_connections 2024-01-15T11:25:16.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.742+0000 7f233f0e6700 1 -- 172.21.15.113:0/2454911307 wait complete. 2024-01-15T11:25:16.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.743+0000 7f233f0e6700 1 Processor -- start 2024-01-15T11:25:16.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.743+0000 7f233f0e6700 1 -- start start 2024-01-15T11:25:16.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233f0e6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2338073f90 0x7f2338145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233f0e6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23380ce090 0x7f2338149150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233f0e6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f233815e2a0 0x7f23380b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233f0e6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2338146430 con 0x7f2338073f90 2024-01-15T11:25:16.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233f0e6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2338149690 con 0x7f23380ce090 2024-01-15T11:25:16.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233f0e6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2338158450 con 0x7f233815e2a0 2024-01-15T11:25:16.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233ce82700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2338073f90 0x7f2338145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233ce82700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2338073f90 0x7f2338145ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45416/0 (socket says 172.21.15.113:45416) 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233ce82700 1 -- 172.21.15.113:0/2269360145 learned_addr learned my addr 172.21.15.113:0/2269360145 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f2337fff700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23380ce090 0x7f2338149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233d683700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f233815e2a0 0x7f23380b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233ce82700 1 -- 172.21.15.113:0/2269360145 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f233815e2a0 msgr2=0x7f23380b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233ce82700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f233815e2a0 0x7f23380b17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233ce82700 1 -- 172.21.15.113:0/2269360145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23380ce090 msgr2=0x7f2338149150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233ce82700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23380ce090 0x7f2338149150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233ce82700 1 -- 172.21.15.113:0/2269360145 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2330009b30 con 0x7f2338073f90 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f2337fff700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23380ce090 0x7f2338149150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233d683700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f233815e2a0 0x7f23380b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f233ce82700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2338073f90 0x7f2338145ef0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f232c00c4d0 tx=0x7f232c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.744+0000 7f23357fa700 1 -- 172.21.15.113:0/2269360145 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c011790 con 0x7f2338073f90 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.745+0000 7f23357fa700 1 -- 172.21.15.113:0/2269360145 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c0118f0 con 0x7f2338073f90 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.745+0000 7f23357fa700 1 -- 172.21.15.113:0/2269360145 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c011ca0 con 0x7f2338073f90 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.745+0000 7f233f0e6700 1 -- 172.21.15.113:0/2269360145 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2338158730 con 0x7f2338073f90 2024-01-15T11:25:16.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.745+0000 7f233f0e6700 1 -- 172.21.15.113:0/2269360145 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2338158cf0 con 0x7f2338073f90 2024-01-15T11:25:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.746+0000 7f233f0e6700 1 -- 172.21.15.113:0/2269360145 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2338067f60 con 0x7f2338073f90 2024-01-15T11:25:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.746+0000 7f23357fa700 1 -- 172.21.15.113:0/2269360145 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f232c01c460 con 0x7f2338073f90 2024-01-15T11:25:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.746+0000 7f23357fa700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f231c06b350 0x7f231c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.746+0000 7f23357fa700 1 -- 172.21.15.113:0/2269360145 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f232c089a00 con 0x7f2338073f90 2024-01-15T11:25:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.746+0000 7f2337fff700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f231c06b350 0x7f231c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.747+0000 7f2337fff700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f231c06b350 0x7f231c06d810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f233815add0 tx=0x7f2324008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:16.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.751+0000 7f23357fa700 1 -- 172.21.15.113:0/2269360145 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f232c054870 con 0x7f2338073f90 2024-01-15T11:25:16.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.942+0000 7f233f0e6700 1 -- 172.21.15.113:0/2269360145 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2338072e00 con 0x7f2338073f90 2024-01-15T11:25:16.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.943+0000 7f23357fa700 1 -- 172.21.15.113:0/2269360145 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16578 (secure 0 0 0) 0x7f232c057e90 con 0x7f2338073f90 2024-01-15T11:25:16.941 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:16.941 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":19,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:15.611278+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:16.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.945+0000 7f233f0e6700 1 -- 172.21.15.113:0/2269360145 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f231c06b350 msgr2=0x7f231c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:16.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.945+0000 7f233f0e6700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f231c06b350 0x7f231c06d810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f233815add0 tx=0x7f2324008040 comp rx=0 tx=0).stop 2024-01-15T11:25:16.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.945+0000 7f233f0e6700 1 -- 172.21.15.113:0/2269360145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2338073f90 msgr2=0x7f2338145ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:16.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.945+0000 7f233f0e6700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2338073f90 0x7f2338145ef0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f232c00c4d0 tx=0x7f232c00c890 comp rx=0 tx=0).stop 2024-01-15T11:25:16.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.945+0000 7f233f0e6700 1 -- 172.21.15.113:0/2269360145 shutdown_connections 2024-01-15T11:25:16.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.945+0000 7f233f0e6700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f231c06b350 0x7f231c06d810 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.945+0000 7f233f0e6700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f233815e2a0 0x7f23380b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.945+0000 7f233f0e6700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2338073f90 0x7f2338145ef0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.945+0000 7f233f0e6700 1 --2- 172.21.15.113:0/2269360145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23380ce090 0x7f2338149150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:16.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.945+0000 7f233f0e6700 1 -- 172.21.15.113:0/2269360145 >> 172.21.15.113:0/2269360145 conn(0x7f23380cd570 msgr2=0x7f23380afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:16.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.945+0000 7f233f0e6700 1 -- 172.21.15.113:0/2269360145 shutdown_connections 2024-01-15T11:25:16.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:16.945+0000 7f233f0e6700 1 -- 172.21.15.113:0/2269360145 wait complete. 2024-01-15T11:25:16.955 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T11:25:17.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.081+0000 7f5ed2688700 1 Processor -- start 2024-01-15T11:25:17.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.081+0000 7f5ed2688700 1 -- start start 2024-01-15T11:25:17.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.082+0000 7f5ed2688700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5ecc0d0a10 0x7f5ecc071400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:17.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.082+0000 7f5ed2688700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5ecc06f0a0 0x7f5ecc071940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.082+0000 7f5ed2688700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc160700 con 0x7f5ecc159aa0 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.082+0000 7f5ed2688700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ecc15faf0 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.082+0000 7f5ed2688700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc15fdd0 con 0x7f5ecc0d0a10 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.082+0000 7f5ed0c25700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5ecc06f0a0 0x7f5ecc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.082+0000 7f5ed0c25700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5ecc06f0a0 0x7f5ecc071940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45420/0 (socket says 172.21.15.113:45420) 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.082+0000 7f5ed0c25700 1 -- 172.21.15.113:0/3229908457 learned_addr learned my addr 172.21.15.113:0/3229908457 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.082+0000 7f5ecb7fe700 1 --2- 172.21.15.113:0/3229908457 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5ecc0d0a10 0x7f5ecc071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.082+0000 7f5ed0c25700 1 -- 172.21.15.113:0/3229908457 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5ecc0d0a10 msgr2=0x7f5ecc071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.082+0000 7f5ed0c25700 1 --2- 172.21.15.113:0/3229908457 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5ecc0d0a10 0x7f5ecc071400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.082+0000 7f5ed0c25700 1 -- 172.21.15.113:0/3229908457 >> v1:172.21.15.141:6789/0 conn(0x7f5ecc159aa0 legacy=0x7f5ecc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ed0c25700 1 -- 172.21.15.113:0/3229908457 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc1600b0 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ecb7fe700 1 --2- 172.21.15.113:0/3229908457 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5ecc0d0a10 0x7f5ecc071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:17.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ed0c25700 1 --2- 172.21.15.113:0/3229908457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc06f0a0 0x7f5ecc071940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009e50 tx=0x7f5ec400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6d21e083e510a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:17.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ecaffd700 1 -- 172.21.15.113:0/3229908457 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec400e070 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ecaffd700 1 -- 172.21.15.113:0/3229908457 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec4003a20 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ecaffd700 1 -- 172.21.15.113:0/3229908457 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec4003dd0 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ed2688700 1 -- 172.21.15.113:0/3229908457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc06f0a0 msgr2=0x7f5ecc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:17.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ed2688700 1 --2- 172.21.15.113:0/3229908457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc06f0a0 0x7f5ecc071940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009e50 tx=0x7f5ec400b330 comp rx=0 tx=0).stop 2024-01-15T11:25:17.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ed2688700 1 -- 172.21.15.113:0/3229908457 shutdown_connections 2024-01-15T11:25:17.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ed2688700 1 --2- 172.21.15.113:0/3229908457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc06f0a0 0x7f5ecc071940 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:17.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ed2688700 1 --2- 172.21.15.113:0/3229908457 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5ecc0d0a10 0x7f5ecc071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:17.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ed2688700 1 -- 172.21.15.113:0/3229908457 >> 172.21.15.113:0/3229908457 conn(0x7f5ecc0743b0 msgr2=0x7f5ecc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:17.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ed2688700 1 -- 172.21.15.113:0/3229908457 shutdown_connections 2024-01-15T11:25:17.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.083+0000 7f5ed2688700 1 -- 172.21.15.113:0/3229908457 wait complete. 2024-01-15T11:25:17.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.084+0000 7f5ed2688700 1 Processor -- start 2024-01-15T11:25:17.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.085+0000 7f5ed2688700 1 -- start start 2024-01-15T11:25:17.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.085+0000 7f5ed2688700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc06f0a0 0x7f5ecc0a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:17.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.085+0000 7f5ed2688700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5ecc0d0a10 0x7f5ecc1581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:17.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.085+0000 7f5ed2688700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5ecc159aa0 0x7f5ecc156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:17.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.085+0000 7f5ed2688700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc1557a0 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.085+0000 7f5ed2688700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc155a80 con 0x7f5ecc159aa0 2024-01-15T11:25:17.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.085+0000 7f5ed2688700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc155d60 con 0x7f5ecc0d0a10 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.085+0000 7f5ecbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc06f0a0 0x7f5ecc0a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ecbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc06f0a0 0x7f5ecc0a9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45422/0 (socket says 172.21.15.113:45422) 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ecbfff700 1 -- 172.21.15.113:0/3770754522 learned_addr learned my addr 172.21.15.113:0/3770754522 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ecb7fe700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5ecc0d0a10 0x7f5ecc1581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ecbfff700 1 -- 172.21.15.113:0/3770754522 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5ecc0d0a10 msgr2=0x7f5ecc1581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ed0c25700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5ecc159aa0 0x7f5ecc156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ecbfff700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5ecc0d0a10 0x7f5ecc1581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ecbfff700 1 -- 172.21.15.113:0/3770754522 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5ecc159aa0 msgr2=0x7f5ecc156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ecbfff700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5ecc159aa0 0x7f5ecc156840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ecbfff700 1 -- 172.21.15.113:0/3770754522 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec4009b30 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ecb7fe700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5ecc0d0a10 0x7f5ecc1581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ecbfff700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc06f0a0 0x7f5ecc0a9e70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000b4a0 tx=0x7f5ec000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ec8ff9700 1 -- 172.21.15.113:0/3770754522 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0011830 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ec8ff9700 1 -- 172.21.15.113:0/3770754522 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec0011990 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ed2688700 1 -- 172.21.15.113:0/3770754522 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc156040 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ec8ff9700 1 -- 172.21.15.113:0/3770754522 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0011d40 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.086+0000 7f5ed2688700 1 -- 172.21.15.113:0/3770754522 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc0aa3b0 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.087+0000 7f5ed2688700 1 -- 172.21.15.113:0/3770754522 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc069020 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.088+0000 7f5ec8ff9700 1 -- 172.21.15.113:0/3770754522 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5ec001a400 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.088+0000 7f5ec8ff9700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f5eb806b350 0x7f5eb806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:17.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.088+0000 7f5ecb7fe700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f5eb806b350 0x7f5eb806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:17.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.088+0000 7f5ec8ff9700 1 -- 172.21.15.113:0/3770754522 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ec00898b0 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.088+0000 7f5ecb7fe700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f5eb806b350 0x7f5eb806d810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc14eb70 tx=0x7f5ebc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:17.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.092+0000 7f5ec8ff9700 1 -- 172.21.15.113:0/3770754522 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ec0054720 con 0x7f5ecc06f0a0 2024-01-15T11:25:17.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:17.329+0000 7f5ed2688700 1 -- 172.21.15.113:0/3770754522 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5ecc06b200 con 0x7f5ecc06f0a0 2024-01-15T11:25:18.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.258+0000 7f5ec8ff9700 1 -- 172.21.15.113:0/3770754522 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5ec00150b0 con 0x7f5ecc06f0a0 2024-01-15T11:25:18.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.261+0000 7f5ed2688700 1 -- 172.21.15.113:0/3770754522 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f5eb806b350 msgr2=0x7f5eb806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:18.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.261+0000 7f5ed2688700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f5eb806b350 0x7f5eb806d810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc14eb70 tx=0x7f5ebc008040 comp rx=0 tx=0).stop 2024-01-15T11:25:18.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.261+0000 7f5ed2688700 1 -- 172.21.15.113:0/3770754522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc06f0a0 msgr2=0x7f5ecc0a9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:18.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.261+0000 7f5ed2688700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc06f0a0 0x7f5ecc0a9e70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000b4a0 tx=0x7f5ec000b860 comp rx=0 tx=0).stop 2024-01-15T11:25:18.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.261+0000 7f5ed2688700 1 -- 172.21.15.113:0/3770754522 shutdown_connections 2024-01-15T11:25:18.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.261+0000 7f5ed2688700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f5eb806b350 0x7f5eb806d810 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:18.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.261+0000 7f5ed2688700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5ecc159aa0 0x7f5ecc156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:18.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.261+0000 7f5ed2688700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc06f0a0 0x7f5ecc0a9e70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:18.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.261+0000 7f5ed2688700 1 --2- 172.21.15.113:0/3770754522 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5ecc0d0a10 0x7f5ecc1581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:18.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.261+0000 7f5ed2688700 1 -- 172.21.15.113:0/3770754522 >> 172.21.15.113:0/3770754522 conn(0x7f5ecc0743b0 msgr2=0x7f5ecc0cf2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:18.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.262+0000 7f5ed2688700 1 -- 172.21.15.113:0/3770754522 shutdown_connections 2024-01-15T11:25:18.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.262+0000 7f5ed2688700 1 -- 172.21.15.113:0/3770754522 wait complete. 2024-01-15T11:25:18.274 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T11:25:18.333 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T11:25:18.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.469+0000 7f1832ba6700 1 Processor -- start 2024-01-15T11:25:18.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.469+0000 7f1832ba6700 1 -- start start 2024-01-15T11:25:18.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.470+0000 7f1832ba6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f182c06f580 0x7f182c15e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:18.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.470+0000 7f1832ba6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f182c15ec50 0x7f182c06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:18.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.470+0000 7f1832ba6700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f182c0b5600 con 0x7f182c158600 2024-01-15T11:25:18.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.470+0000 7f1832ba6700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182c15f0e0 con 0x7f182c15ec50 2024-01-15T11:25:18.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.470+0000 7f1832ba6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182c06fa10 con 0x7f182c06f580 2024-01-15T11:25:18.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.470+0000 7f1831143700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f182c15ec50 0x7f182c06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.470+0000 7f1831143700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f182c15ec50 0x7f182c06aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45436/0 (socket says 172.21.15.113:45436) 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.470+0000 7f1831143700 1 -- 172.21.15.113:0/206318402 learned_addr learned my addr 172.21.15.113:0/206318402 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.470+0000 7f1831143700 1 -- 172.21.15.113:0/206318402 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f182c06f580 msgr2=0x7f182c15e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.470+0000 7f1831143700 1 --2- 172.21.15.113:0/206318402 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f182c06f580 0x7f182c15e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.470+0000 7f1831143700 1 -- 172.21.15.113:0/206318402 >> v1:172.21.15.141:6789/0 conn(0x7f182c158600 legacy=0x7f182c0b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.470+0000 7f1831143700 1 -- 172.21.15.113:0/206318402 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182c06e4e0 con 0x7f182c15ec50 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.471+0000 7f1831143700 1 --2- 172.21.15.113:0/206318402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f182c15ec50 0x7f182c06aff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f182400d150 tx=0x7f182400d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d56feaa5fb6c4f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.471+0000 7f182b7fe700 1 -- 172.21.15.113:0/206318402 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824010070 con 0x7f182c15ec50 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.471+0000 7f182b7fe700 1 -- 172.21.15.113:0/206318402 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182400dd80 con 0x7f182c15ec50 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.471+0000 7f182b7fe700 1 -- 172.21.15.113:0/206318402 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182400b310 con 0x7f182c15ec50 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.471+0000 7f1832ba6700 1 -- 172.21.15.113:0/206318402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f182c15ec50 msgr2=0x7f182c06aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.471+0000 7f1832ba6700 1 --2- 172.21.15.113:0/206318402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f182c15ec50 0x7f182c06aff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f182400d150 tx=0x7f182400d460 comp rx=0 tx=0).stop 2024-01-15T11:25:18.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.471+0000 7f1832ba6700 1 -- 172.21.15.113:0/206318402 shutdown_connections 2024-01-15T11:25:18.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.471+0000 7f1832ba6700 1 --2- 172.21.15.113:0/206318402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f182c15ec50 0x7f182c06aff0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:18.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.471+0000 7f1832ba6700 1 --2- 172.21.15.113:0/206318402 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f182c06f580 0x7f182c15e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:18.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.471+0000 7f1832ba6700 1 -- 172.21.15.113:0/206318402 >> 172.21.15.113:0/206318402 conn(0x7f182c0b3250 msgr2=0x7f182c0d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:18.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.471+0000 7f1832ba6700 1 -- 172.21.15.113:0/206318402 shutdown_connections 2024-01-15T11:25:18.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.471+0000 7f1832ba6700 1 -- 172.21.15.113:0/206318402 wait complete. 2024-01-15T11:25:18.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.472+0000 7f1832ba6700 1 Processor -- start 2024-01-15T11:25:18.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.472+0000 7f1832ba6700 1 -- start start 2024-01-15T11:25:18.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.473+0000 7f1832ba6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f182c06f580 0x7f182c156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:18.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.473+0000 7f1832ba6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f182c158600 0x7f182c1566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:18.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.473+0000 7f1832ba6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f182c15ec50 0x7f182c09dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.473+0000 7f1832ba6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c09bd50 con 0x7f182c158600 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.473+0000 7f1832ba6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c0d06d0 con 0x7f182c06f580 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.473+0000 7f1832ba6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182c14b710 con 0x7f182c15ec50 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.473+0000 7f182bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f182c158600 0x7f182c1566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.473+0000 7f182bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f182c158600 0x7f182c1566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45444/0 (socket says 172.21.15.113:45444) 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.473+0000 7f182bfff700 1 -- 172.21.15.113:0/229273406 learned_addr learned my addr 172.21.15.113:0/229273406 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f182bfff700 1 -- 172.21.15.113:0/229273406 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f182c15ec50 msgr2=0x7f182c09dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f1830942700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f182c06f580 0x7f182c156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f1831143700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f182c15ec50 0x7f182c09dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f182bfff700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f182c15ec50 0x7f182c09dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f182bfff700 1 -- 172.21.15.113:0/229273406 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f182c06f580 msgr2=0x7f182c156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f182bfff700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f182c06f580 0x7f182c156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f182bfff700 1 -- 172.21.15.113:0/229273406 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182400bd40 con 0x7f182c158600 2024-01-15T11:25:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f1830942700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f182c06f580 0x7f182c156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:18.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f1831143700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f182c15ec50 0x7f182c09dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:18.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f182bfff700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f182c158600 0x7f182c1566c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f181800b530 tx=0x7f181800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:18.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f18297fa700 1 -- 172.21.15.113:0/229273406 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818011830 con 0x7f182c158600 2024-01-15T11:25:18.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f18297fa700 1 -- 172.21.15.113:0/229273406 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18180119b0 con 0x7f182c158600 2024-01-15T11:25:18.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f1832ba6700 1 -- 172.21.15.113:0/229273406 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182c09b0d0 con 0x7f182c158600 2024-01-15T11:25:18.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f18297fa700 1 -- 172.21.15.113:0/229273406 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818011d60 con 0x7f182c158600 2024-01-15T11:25:18.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.474+0000 7f1832ba6700 1 -- 172.21.15.113:0/229273406 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182c09b690 con 0x7f182c158600 2024-01-15T11:25:18.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.475+0000 7f1832ba6700 1 -- 172.21.15.113:0/229273406 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182c09be90 con 0x7f182c158600 2024-01-15T11:25:18.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.476+0000 7f18297fa700 1 -- 172.21.15.113:0/229273406 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f181801a400 con 0x7f182c158600 2024-01-15T11:25:18.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.476+0000 7f18297fa700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f181006b350 0x7f181006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:18.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.476+0000 7f18297fa700 1 -- 172.21.15.113:0/229273406 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1818089f60 con 0x7f182c158600 2024-01-15T11:25:18.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.476+0000 7f1830942700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f181006b350 0x7f181006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:18.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.477+0000 7f1830942700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f181006b350 0x7f181006d810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1820001750 tx=0x7f1820009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:18.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.480+0000 7f18297fa700 1 -- 172.21.15.113:0/229273406 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1818054f40 con 0x7f182c158600 2024-01-15T11:25:18.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:18.718+0000 7f1832ba6700 1 -- 172.21.15.113:0/229273406 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f182c09be90 con 0x7f182c158600 2024-01-15T11:25:19.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.265+0000 7f18297fa700 1 -- 172.21.15.113:0/229273406 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f182c09be90 con 0x7f182c158600 2024-01-15T11:25:19.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.268+0000 7f1832ba6700 1 -- 172.21.15.113:0/229273406 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f181006b350 msgr2=0x7f181006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.268+0000 7f1832ba6700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f181006b350 0x7f181006d810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1820001750 tx=0x7f1820009040 comp rx=0 tx=0).stop 2024-01-15T11:25:19.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.268+0000 7f1832ba6700 1 -- 172.21.15.113:0/229273406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f182c158600 msgr2=0x7f182c1566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.268+0000 7f1832ba6700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f182c158600 0x7f182c1566c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f181800b530 tx=0x7f181800b8f0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.268+0000 7f1832ba6700 1 -- 172.21.15.113:0/229273406 shutdown_connections 2024-01-15T11:25:19.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.268+0000 7f1832ba6700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f181006b350 0x7f181006d810 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.268+0000 7f1832ba6700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f182c15ec50 0x7f182c09dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.268+0000 7f1832ba6700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f182c06f580 0x7f182c156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.268+0000 7f1832ba6700 1 --2- 172.21.15.113:0/229273406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f182c158600 0x7f182c1566c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.268+0000 7f1832ba6700 1 -- 172.21.15.113:0/229273406 >> 172.21.15.113:0/229273406 conn(0x7f182c0b3250 msgr2=0x7f182c158c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:19.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.268+0000 7f1832ba6700 1 -- 172.21.15.113:0/229273406 shutdown_connections 2024-01-15T11:25:19.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.268+0000 7f1832ba6700 1 -- 172.21.15.113:0/229273406 wait complete. 2024-01-15T11:25:19.279 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T11:25:19.279 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T11:25:19.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.403+0000 7fc6e77f4700 1 Processor -- start 2024-01-15T11:25:19.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.403+0000 7fc6e77f4700 1 -- start start 2024-01-15T11:25:19.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6e77f4700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e015bd40 0x7fc6e00cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6e77f4700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc6e0071340 0x7fc6e00717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6e77f4700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e006d540 con 0x7fc6e00afb20 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6e77f4700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e0071cf0 con 0x7fc6e0071340 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6e77f4700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e006d820 con 0x7fc6e015bd40 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6e4d8f700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fc6e00afb20 0x7fc6e015c120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:37442/0 (socket says 172.21.15.113:37442) 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6e4d8f700 1 -- 172.21.15.113:0/3264091844 learned_addr learned my addr 172.21.15.113:0/3264091844 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062673615 0 0) 0x7fc6e006d540 con 0x7fc6e00afb20 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6cc0035b0 con 0x7fc6e00afb20 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6e5d91700 1 --2- 172.21.15.113:0/3264091844 >> v2:172.21.15.141:3300/0 conn(0x7fc6e0071340 0x7fc6e00717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6e5590700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e015bd40 0x7fc6e00cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2179737368 0 0) 0x7fc6cc0035b0 con 0x7fc6e00afb20 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e015bd40 msgr2=0x7fc6e00cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e015bd40 0x7fc6e00cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 >> v2:172.21.15.141:3300/0 conn(0x7fc6e0071340 msgr2=0x7fc6e00717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:19.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 --2- 172.21.15.113:0/3264091844 >> v2:172.21.15.141:3300/0 conn(0x7fc6e0071340 0x7fc6e00717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e006db00 con 0x7fc6e00afb20 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fc6d0001670 con 0x7fc6e00afb20 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6e5590700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e015bd40 0x7fc6e00cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 >> v1:172.21.15.113:6789/0 conn(0x7fc6e00afb20 legacy=0x7fc6e015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e015bd40 0x7fc6cc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6cc003f40 0x7fc6cc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6cc004d90 0x7fc6cc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e0071cf0 con 0x7fc6cc003f40 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e006d820 con 0x7fc6cc004d90 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.404+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc00ba40 con 0x7fc6e015bd40 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e4d8f700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6cc003f40 0x7fc6cc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e5590700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e015bd40 0x7fc6cc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e5d91700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6cc004d90 0x7fc6cc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e4d8f700 1 -- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e015bd40 msgr2=0x7fc6cc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e4d8f700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e015bd40 0x7fc6cc003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e4d8f700 1 -- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6cc004d90 msgr2=0x7fc6cc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e4d8f700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6cc004d90 0x7fc6cc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e4d8f700 1 -- 172.21.15.113:0/3264091844 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc0035b0 con 0x7fc6cc003f40 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e5d91700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6cc004d90 0x7fc6cc005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e4d8f700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6cc003f40 0x7fc6cc004850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000c2f0 tx=0x7fc6cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0b8e16250f1206 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d000ce30 con 0x7fc6cc003f40 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d0005960 con 0x7fc6cc003f40 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6d7fff700 1 -- 172.21.15.113:0/3264091844 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0005d10 con 0x7fc6cc003f40 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e77f4700 1 -- 172.21.15.113:0/3264091844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6cc003f40 msgr2=0x7fc6cc004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e77f4700 1 --2- 172.21.15.113:0/3264091844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6cc003f40 0x7fc6cc004850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000c2f0 tx=0x7fc6cc005880 comp rx=0 tx=0).stop 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.405+0000 7fc6e5d91700 1 -- 172.21.15.113:0/3264091844 reap_dead start 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.406+0000 7fc6e77f4700 1 -- 172.21.15.113:0/3264091844 shutdown_connections 2024-01-15T11:25:19.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.406+0000 7fc6e77f4700 1 -- 172.21.15.113:0/3264091844 >> 172.21.15.113:0/3264091844 conn(0x7fc6e00743b0 msgr2=0x7fc6e006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.406+0000 7fc6e77f4700 1 -- 172.21.15.113:0/3264091844 shutdown_connections 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.406+0000 7fc6e77f4700 1 -- 172.21.15.113:0/3264091844 wait complete. 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.407+0000 7fc6e77f4700 1 Processor -- start 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.407+0000 7fc6e77f4700 1 -- start start 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e77f4700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6cc003f40 0x7fc6e0152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e77f4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6e0149c90 0x7fc6e0155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e77f4700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e00ad870 0x7fc6e00adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e77f4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e0153ed0 con 0x7fc6e0149c90 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e77f4700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e00ae230 con 0x7fc6cc003f40 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e77f4700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e01542f0 con 0x7fc6e00ad870 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e4d8f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6e0149c90 0x7fc6e0155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e4d8f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6e0149c90 0x7fc6e0155110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45462/0 (socket says 172.21.15.113:45462) 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e4d8f700 1 -- 172.21.15.113:0/366281003 learned_addr learned my addr 172.21.15.113:0/366281003 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:19.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e5d91700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e00ad870 0x7fc6e00adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e5590700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6cc003f40 0x7fc6e0152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e4d8f700 1 -- 172.21.15.113:0/366281003 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e00ad870 msgr2=0x7fc6e00adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e4d8f700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e00ad870 0x7fc6e00adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e4d8f700 1 -- 172.21.15.113:0/366281003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6cc003f40 msgr2=0x7fc6e0152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e4d8f700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6cc003f40 0x7fc6e0152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e4d8f700 1 -- 172.21.15.113:0/366281003 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e0155650 con 0x7fc6e0149c90 2024-01-15T11:25:19.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e4d8f700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6e0149c90 0x7fc6e0155110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0000c00 tx=0x7fc6d0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:19.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6d5ffb700 1 -- 172.21.15.113:0/366281003 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0001660 con 0x7fc6e0149c90 2024-01-15T11:25:19.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6d5ffb700 1 -- 172.21.15.113:0/366281003 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d0004a80 con 0x7fc6e0149c90 2024-01-15T11:25:19.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e77f4700 1 -- 172.21.15.113:0/366281003 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e0153250 con 0x7fc6e0149c90 2024-01-15T11:25:19.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6d5ffb700 1 -- 172.21.15.113:0/366281003 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0004e30 con 0x7fc6e0149c90 2024-01-15T11:25:19.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.408+0000 7fc6e77f4700 1 -- 172.21.15.113:0/366281003 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e0153810 con 0x7fc6e0149c90 2024-01-15T11:25:19.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.409+0000 7fc6e77f4700 1 -- 172.21.15.113:0/366281003 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e00753f0 con 0x7fc6e0149c90 2024-01-15T11:25:19.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.410+0000 7fc6d5ffb700 1 -- 172.21.15.113:0/366281003 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc6d00070d0 con 0x7fc6e0149c90 2024-01-15T11:25:19.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.410+0000 7fc6d5ffb700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc6c406b380 0x7fc6c406d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.410+0000 7fc6e5590700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc6c406b380 0x7fc6c406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.410+0000 7fc6d5ffb700 1 -- 172.21.15.113:0/366281003 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6d008af90 con 0x7fc6e0149c90 2024-01-15T11:25:19.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.411+0000 7fc6e5590700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc6c406b380 0x7fc6c406d840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc004410 tx=0x7fc6dc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:19.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.415+0000 7fc6d5ffb700 1 -- 172.21.15.113:0/366281003 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6d00bde90 con 0x7fc6e0149c90 2024-01-15T11:25:19.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.646+0000 7fc6e77f4700 1 -- 172.21.15.113:0/366281003 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6e0154010 con 0x7fc6e0149c90 2024-01-15T11:25:19.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.647+0000 7fc6d5ffb700 1 -- 172.21.15.113:0/366281003 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7fc6e0154010 con 0x7fc6e0149c90 2024-01-15T11:25:19.646 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:19.646 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3164513953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":3164513953},{"type":"v1","addr":"172.21.15.113:6835","nonce":3164513953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/891522731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":891522731},{"type":"v1","addr":"172.21.15.113:6837","nonce":891522731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2158560535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2158560535},{"type":"v1","addr":"172.21.15.141:6834","nonce":2158560535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T11:25:15.611582+0000","modified":"2024-01-15T11:25:18.718623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4300":{"gid":4300,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1012738257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1012738257},{"type":"v1","addr":"172.21.15.141:6835","nonce":1012738257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T11:25:19.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.649+0000 7fc6e77f4700 1 -- 172.21.15.113:0/366281003 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc6c406b380 msgr2=0x7fc6c406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.649+0000 7fc6e77f4700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc6c406b380 0x7fc6c406d840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc004410 tx=0x7fc6dc008040 comp rx=0 tx=0).stop 2024-01-15T11:25:19.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.649+0000 7fc6e77f4700 1 -- 172.21.15.113:0/366281003 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6e0149c90 msgr2=0x7fc6e0155110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.649+0000 7fc6e77f4700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6e0149c90 0x7fc6e0155110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0000c00 tx=0x7fc6d0001a00 comp rx=0 tx=0).stop 2024-01-15T11:25:19.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.649+0000 7fc6e77f4700 1 -- 172.21.15.113:0/366281003 shutdown_connections 2024-01-15T11:25:19.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.649+0000 7fc6e77f4700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc6c406b380 0x7fc6c406d840 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.649+0000 7fc6e77f4700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6e00ad870 0x7fc6e00adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.649+0000 7fc6e77f4700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6cc003f40 0x7fc6e0152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.649+0000 7fc6e77f4700 1 --2- 172.21.15.113:0/366281003 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6e0149c90 0x7fc6e0155110 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.649+0000 7fc6e77f4700 1 -- 172.21.15.113:0/366281003 >> 172.21.15.113:0/366281003 conn(0x7fc6e00743b0 msgr2=0x7fc6e00b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:19.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.649+0000 7fc6e77f4700 1 -- 172.21.15.113:0/366281003 shutdown_connections 2024-01-15T11:25:19.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.649+0000 7fc6e77f4700 1 -- 172.21.15.113:0/366281003 wait complete. 2024-01-15T11:25:19.649 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-15T11:25:19.660 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:19.664 INFO:tasks.ceph.mds.d.smithi141.stderr:2024-01-15T11:25:19.665+0000 7f8d59a92700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T11:25:19.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.782+0000 7f0ec131f700 1 Processor -- start 2024-01-15T11:25:19.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.782+0000 7f0ec131f700 1 -- start start 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ec131f700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0ebc15e090 0x7f0ebc06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ec131f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0ebc0d2250 0x7f0ebc0b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ec131f700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc0d0ee0 con 0x7f0ebc06c090 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ec131f700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ebc0d02d0 con 0x7f0ebc0d2250 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ec131f700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc0d05b0 con 0x7f0ebc15e090 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ebb7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0ebc0d2250 0x7f0ebc0b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ebb7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0ebc0d2250 0x7f0ebc0b15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45472/0 (socket says 172.21.15.113:45472) 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ebb7fe700 1 -- 172.21.15.113:0/2954937977 learned_addr learned my addr 172.21.15.113:0/2954937977 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ebaffd700 1 --2- 172.21.15.113:0/2954937977 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0ebc15e090 0x7f0ebc06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ebb7fe700 1 -- 172.21.15.113:0/2954937977 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0ebc15e090 msgr2=0x7f0ebc06bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ebb7fe700 1 --2- 172.21.15.113:0/2954937977 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0ebc15e090 0x7f0ebc06bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ebb7fe700 1 -- 172.21.15.113:0/2954937977 >> v1:172.21.15.141:6789/0 conn(0x7f0ebc06c090 legacy=0x7f0ebc0d1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:19.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ebb7fe700 1 -- 172.21.15.113:0/2954937977 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc0d0890 con 0x7f0ebc0d2250 2024-01-15T11:25:19.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0ebb7fe700 1 --2- 172.21.15.113:0/2954937977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ebc0d2250 0x7f0ebc0b15c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4009e50 tx=0x7f0eb400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c7a3b64a7d243a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:19.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0eb9ffb700 1 -- 172.21.15.113:0/2954937977 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb400e070 con 0x7f0ebc0d2250 2024-01-15T11:25:19.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0eb9ffb700 1 -- 172.21.15.113:0/2954937977 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb4003a20 con 0x7f0ebc0d2250 2024-01-15T11:25:19.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.783+0000 7f0eb9ffb700 1 -- 172.21.15.113:0/2954937977 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4003dd0 con 0x7f0ebc0d2250 2024-01-15T11:25:19.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.784+0000 7f0ec131f700 1 -- 172.21.15.113:0/2954937977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ebc0d2250 msgr2=0x7f0ebc0b15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.784+0000 7f0ec131f700 1 --2- 172.21.15.113:0/2954937977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ebc0d2250 0x7f0ebc0b15c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4009e50 tx=0x7f0eb400b330 comp rx=0 tx=0).stop 2024-01-15T11:25:19.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.784+0000 7f0ec131f700 1 -- 172.21.15.113:0/2954937977 shutdown_connections 2024-01-15T11:25:19.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.784+0000 7f0ec131f700 1 --2- 172.21.15.113:0/2954937977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ebc0d2250 0x7f0ebc0b15c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.784+0000 7f0ec131f700 1 --2- 172.21.15.113:0/2954937977 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0ebc15e090 0x7f0ebc06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.784+0000 7f0ec131f700 1 -- 172.21.15.113:0/2954937977 >> 172.21.15.113:0/2954937977 conn(0x7f0ebc0731a0 msgr2=0x7f0ebc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:19.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.784+0000 7f0ec131f700 1 -- 172.21.15.113:0/2954937977 shutdown_connections 2024-01-15T11:25:19.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.784+0000 7f0ec131f700 1 -- 172.21.15.113:0/2954937977 wait complete. 2024-01-15T11:25:19.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.785+0000 7f0ec131f700 1 Processor -- start 2024-01-15T11:25:19.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.786+0000 7f0ec131f700 1 -- start start 2024-01-15T11:25:19.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.786+0000 7f0ec131f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ebc06c090 0x7f0ebc15a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.786+0000 7f0ec131f700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ebc0d2250 0x7f0ebc0cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.786+0000 7f0ec131f700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0ebc15e090 0x7f0ebc0b7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.786+0000 7f0ec131f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc0c6840 con 0x7f0ebc06c090 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.786+0000 7f0ec131f700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc0c6ae0 con 0x7f0ebc0d2250 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.786+0000 7f0ec131f700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc0c6dc0 con 0x7f0ebc15e090 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ebaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ebc06c090 0x7f0ebc15a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ebaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ebc06c090 0x7f0ebc15a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45474/0 (socket says 172.21.15.113:45474) 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ebaffd700 1 -- 172.21.15.113:0/3481868703 learned_addr learned my addr 172.21.15.113:0/3481868703 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0eba7fc700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ebc0d2250 0x7f0ebc0cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ebb7fe700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0ebc15e090 0x7f0ebc0b7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ebaffd700 1 -- 172.21.15.113:0/3481868703 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0ebc15e090 msgr2=0x7f0ebc0b7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ebaffd700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0ebc15e090 0x7f0ebc0b7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ebaffd700 1 -- 172.21.15.113:0/3481868703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ebc0d2250 msgr2=0x7f0ebc0cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ebaffd700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ebc0d2250 0x7f0ebc0cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ebaffd700 1 -- 172.21.15.113:0/3481868703 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb4009b30 con 0x7f0ebc06c090 2024-01-15T11:25:19.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ebaffd700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ebc06c090 0x7f0ebc15a170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00c7f0 tx=0x7f0eac00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:19.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ea37fe700 1 -- 172.21.15.113:0/3481868703 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac011800 con 0x7f0ebc06c090 2024-01-15T11:25:19.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ea37fe700 1 -- 172.21.15.113:0/3481868703 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac011960 con 0x7f0ebc06c090 2024-01-15T11:25:19.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ea37fe700 1 -- 172.21.15.113:0/3481868703 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac011cf0 con 0x7f0ebc06c090 2024-01-15T11:25:19.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ec131f700 1 -- 172.21.15.113:0/3481868703 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ebc0c70a0 con 0x7f0ebc06c090 2024-01-15T11:25:19.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.787+0000 7f0ec131f700 1 -- 172.21.15.113:0/3481868703 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ebc0b7870 con 0x7f0ebc06c090 2024-01-15T11:25:19.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.788+0000 7f0ec131f700 1 -- 172.21.15.113:0/3481868703 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ebc070b40 con 0x7f0ebc06c090 2024-01-15T11:25:19.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.788+0000 7f0ea37fe700 1 -- 172.21.15.113:0/3481868703 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0eac018360 con 0x7f0ebc06c090 2024-01-15T11:25:19.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.789+0000 7f0ea37fe700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0ea806b350 0x7f0ea806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:19.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.789+0000 7f0eba7fc700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0ea806b350 0x7f0ea806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:19.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.789+0000 7f0ea37fe700 1 -- 172.21.15.113:0/3481868703 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0eac089b20 con 0x7f0ebc06c090 2024-01-15T11:25:19.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.789+0000 7f0eba7fc700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0ea806b350 0x7f0ea806d810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0cf090 tx=0x7f0eb0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:19.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.793+0000 7f0ea37fe700 1 -- 172.21.15.113:0/3481868703 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0eac054990 con 0x7f0ebc06c090 2024-01-15T11:25:19.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.984+0000 7f0ec131f700 1 -- 172.21.15.113:0/3481868703 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ebc0751e0 con 0x7f0ebc06c090 2024-01-15T11:25:19.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.985+0000 7f0ea37fe700 1 -- 172.21.15.113:0/3481868703 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f0eac01f090 con 0x7f0ebc06c090 2024-01-15T11:25:19.984 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:19.984 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:19.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.987+0000 7f0ec131f700 1 -- 172.21.15.113:0/3481868703 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0ea806b350 msgr2=0x7f0ea806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.987+0000 7f0ec131f700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0ea806b350 0x7f0ea806d810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0cf090 tx=0x7f0eb0009040 comp rx=0 tx=0).stop 2024-01-15T11:25:19.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.987+0000 7f0ec131f700 1 -- 172.21.15.113:0/3481868703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ebc06c090 msgr2=0x7f0ebc15a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:19.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.987+0000 7f0ec131f700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ebc06c090 0x7f0ebc15a170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00c7f0 tx=0x7f0eac00cbb0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.987+0000 7f0ec131f700 1 -- 172.21.15.113:0/3481868703 shutdown_connections 2024-01-15T11:25:19.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.987+0000 7f0ec131f700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0ea806b350 0x7f0ea806d810 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.987+0000 7f0ec131f700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0ebc15e090 0x7f0ebc0b7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.987+0000 7f0ec131f700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ebc06c090 0x7f0ebc15a170 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.987+0000 7f0ec131f700 1 --2- 172.21.15.113:0/3481868703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0ebc0d2250 0x7f0ebc0cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:19.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.987+0000 7f0ec131f700 1 -- 172.21.15.113:0/3481868703 >> 172.21.15.113:0/3481868703 conn(0x7f0ebc0731a0 msgr2=0x7f0ebc0700f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:19.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.987+0000 7f0ec131f700 1 -- 172.21.15.113:0/3481868703 shutdown_connections 2024-01-15T11:25:19.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:19.988+0000 7f0ec131f700 1 -- 172.21.15.113:0/3481868703 wait complete. 2024-01-15T11:25:19.998 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T11:25:19.999 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T11:25:20.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.120+0000 7fe77ffb6700 1 Processor -- start 2024-01-15T11:25:20.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.120+0000 7fe77ffb6700 1 -- start start 2024-01-15T11:25:20.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77ffb6700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe77815af10 0x7fe77815e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77ffb6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe77815f630 0x7fe77806b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77ffb6700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7781414c0 con 0x7fe7780ce140 2024-01-15T11:25:20.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77ffb6700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77815c300 con 0x7fe77815f630 2024-01-15T11:25:20.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77ffb6700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7780ce9f0 con 0x7fe77815af10 2024-01-15T11:25:20.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77e553700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe77815f630 0x7fe77806b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77e553700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe77815f630 0x7fe77806b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45486/0 (socket says 172.21.15.113:45486) 2024-01-15T11:25:20.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77e553700 1 -- 172.21.15.113:0/3346723575 learned_addr learned my addr 172.21.15.113:0/3346723575 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:20.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77dd52700 1 --2- 172.21.15.113:0/3346723575 >> v2:172.21.15.141:3300/0 conn(0x7fe77815af10 0x7fe77815e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77e553700 1 -- 172.21.15.113:0/3346723575 >> v2:172.21.15.141:3300/0 conn(0x7fe77815af10 msgr2=0x7fe77815e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:20.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77e553700 1 --2- 172.21.15.113:0/3346723575 >> v2:172.21.15.141:3300/0 conn(0x7fe77815af10 0x7fe77815e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77e553700 1 -- 172.21.15.113:0/3346723575 >> v1:172.21.15.141:6789/0 conn(0x7fe7780ce140 legacy=0x7fe77815ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77e553700 1 -- 172.21.15.113:0/3346723575 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7780cecd0 con 0x7fe77815f630 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77e553700 1 --2- 172.21.15.113:0/3346723575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe77815f630 0x7fe77806b640 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe77806d0c0 tx=0x7fe76800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad4ae11968eeaf9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77cd50700 1 -- 172.21.15.113:0/3346723575 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76800e070 con 0x7fe77815f630 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77cd50700 1 -- 172.21.15.113:0/3346723575 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768003a20 con 0x7fe77815f630 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.121+0000 7fe77cd50700 1 -- 172.21.15.113:0/3346723575 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768003dd0 con 0x7fe77815f630 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.122+0000 7fe77ffb6700 1 -- 172.21.15.113:0/3346723575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe77815f630 msgr2=0x7fe77806b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.122+0000 7fe77ffb6700 1 --2- 172.21.15.113:0/3346723575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe77815f630 0x7fe77806b640 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe77806d0c0 tx=0x7fe76800b330 comp rx=0 tx=0).stop 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.122+0000 7fe77ffb6700 1 -- 172.21.15.113:0/3346723575 shutdown_connections 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.122+0000 7fe77ffb6700 1 --2- 172.21.15.113:0/3346723575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe77815f630 0x7fe77806b640 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.122+0000 7fe77ffb6700 1 --2- 172.21.15.113:0/3346723575 >> v2:172.21.15.141:3300/0 conn(0x7fe77815af10 0x7fe77815e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.122+0000 7fe77ffb6700 1 -- 172.21.15.113:0/3346723575 >> 172.21.15.113:0/3346723575 conn(0x7fe7780cd570 msgr2=0x7fe7780cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.122+0000 7fe77ffb6700 1 -- 172.21.15.113:0/3346723575 shutdown_connections 2024-01-15T11:25:20.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.122+0000 7fe77ffb6700 1 -- 172.21.15.113:0/3346723575 wait complete. 2024-01-15T11:25:20.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.123+0000 7fe77ffb6700 1 Processor -- start 2024-01-15T11:25:20.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.123+0000 7fe77ffb6700 1 -- start start 2024-01-15T11:25:20.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.124+0000 7fe77ffb6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe7780ce140 0x7fe77813d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.124+0000 7fe77ffb6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe77815af10 0x7fe778159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.124+0000 7fe77ffb6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7781e69a0 0x7fe7781e6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.124+0000 7fe77ffb6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7781e7390 con 0x7fe77815af10 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.124+0000 7fe77ffb6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77815d1d0 con 0x7fe7781e69a0 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.124+0000 7fe77ffb6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77815d4b0 con 0x7fe7780ce140 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.124+0000 7fe77d551700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe77815af10 0x7fe778159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.124+0000 7fe77d551700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe77815af10 0x7fe778159550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45494/0 (socket says 172.21.15.113:45494) 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.124+0000 7fe77d551700 1 -- 172.21.15.113:0/1400065968 learned_addr learned my addr 172.21.15.113:0/1400065968 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe77dd52700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe7780ce140 0x7fe77813d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe77e553700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7781e69a0 0x7fe7781e6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe77d551700 1 -- 172.21.15.113:0/1400065968 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe7780ce140 msgr2=0x7fe77813d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe77d551700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe7780ce140 0x7fe77813d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe77d551700 1 -- 172.21.15.113:0/1400065968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7781e69a0 msgr2=0x7fe7781e6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe77d551700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7781e69a0 0x7fe7781e6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe77d551700 1 -- 172.21.15.113:0/1400065968 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768009b30 con 0x7fe77815af10 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe77d551700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe77815af10 0x7fe778159550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00c4a0 tx=0x7fe76c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe7767fc700 1 -- 172.21.15.113:0/1400065968 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c0054e0 con 0x7fe77815af10 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe7767fc700 1 -- 172.21.15.113:0/1400065968 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76c003680 con 0x7fe77815af10 2024-01-15T11:25:20.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe77ffb6700 1 -- 172.21.15.113:0/1400065968 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77815d790 con 0x7fe77815af10 2024-01-15T11:25:20.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe7767fc700 1 -- 172.21.15.113:0/1400065968 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c003a10 con 0x7fe77815af10 2024-01-15T11:25:20.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.125+0000 7fe77ffb6700 1 -- 172.21.15.113:0/1400065968 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe778142660 con 0x7fe77815af10 2024-01-15T11:25:20.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.126+0000 7fe77ffb6700 1 -- 172.21.15.113:0/1400065968 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7781424c0 con 0x7fe77815af10 2024-01-15T11:25:20.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.126+0000 7fe7767fc700 1 -- 172.21.15.113:0/1400065968 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe76c003b70 con 0x7fe77815af10 2024-01-15T11:25:20.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.127+0000 7fe7767fc700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe76406b350 0x7fe76406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.127+0000 7fe77dd52700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe76406b350 0x7fe76406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.127+0000 7fe7767fc700 1 -- 172.21.15.113:0/1400065968 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe76c088fb0 con 0x7fe77815af10 2024-01-15T11:25:20.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.127+0000 7fe77dd52700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe76406b350 0x7fe76406d810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe77815be60 tx=0x7fe770008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:20.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.131+0000 7fe7767fc700 1 -- 172.21.15.113:0/1400065968 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7781424c0 con 0x7fe77815af10 2024-01-15T11:25:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.324+0000 7fe77ffb6700 1 -- 172.21.15.113:0/1400065968 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe7780cefb0 con 0x7fe76406b350 2024-01-15T11:25:20.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.332+0000 7fe7767fc700 1 -- 172.21.15.113:0/1400065968 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264603 (secure 0 0 0) 0x7fe77000c040 con 0x7fe76406b350 2024-01-15T11:25:20.334 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:20.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.335+0000 7fe77ffb6700 1 -- 172.21.15.113:0/1400065968 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe76406b350 msgr2=0x7fe76406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.335+0000 7fe77ffb6700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe76406b350 0x7fe76406d810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe77815be60 tx=0x7fe770008040 comp rx=0 tx=0).stop 2024-01-15T11:25:20.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.335+0000 7fe77ffb6700 1 -- 172.21.15.113:0/1400065968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe77815af10 msgr2=0x7fe778159550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.335+0000 7fe77ffb6700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe77815af10 0x7fe778159550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00c4a0 tx=0x7fe76c00c860 comp rx=0 tx=0).stop 2024-01-15T11:25:20.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.335+0000 7fe77ffb6700 1 -- 172.21.15.113:0/1400065968 shutdown_connections 2024-01-15T11:25:20.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.335+0000 7fe77ffb6700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe76406b350 0x7fe76406d810 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.335+0000 7fe77ffb6700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe7781e69a0 0x7fe7781e6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.335+0000 7fe77ffb6700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe7780ce140 0x7fe77813d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.335+0000 7fe77ffb6700 1 --2- 172.21.15.113:0/1400065968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe77815af10 0x7fe778159550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.335+0000 7fe77ffb6700 1 -- 172.21.15.113:0/1400065968 >> 172.21.15.113:0/1400065968 conn(0x7fe7780cd570 msgr2=0x7fe7780b4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:20.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.335+0000 7fe77ffb6700 1 -- 172.21.15.113:0/1400065968 shutdown_connections 2024-01-15T11:25:20.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.335+0000 7fe77ffb6700 1 -- 172.21.15.113:0/1400065968 wait complete. 2024-01-15T11:25:20.337 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-15T11:25:20.346 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-15T11:25:20.109581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218888,"kb_used_data":2704,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755832,"statfs":{"total":773094113280,"available":772869971968,"internally_reserved":0,"allocated":2768896,"data_stored":1695023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.894917"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.273391+0000","last_change":"2024-01-15T11:25:16.269790+0000","last_active":"2024-01-15T11:25:16.273391+0000","last_peered":"2024-01-15T11:25:16.273391+0000","last_clean":"2024-01-15T11:25:16.273391+0000","last_became_active":"2024-01-15T11:25:16.269540+0000","last_became_peered":"2024-01-15T11:25:16.269540+0000","last_unstale":"2024-01-15T11:25:16.273391+0000","last_undegraded":"2024-01-15T11:25:16.273391+0000","last_fullsized":"2024-01-15T11:25:16.273391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:02:52.680406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260233+0000","last_change":"2024-01-15T11:25:16.254768+0000","last_active":"2024-01-15T11:25:16.260233+0000","last_peered":"2024-01-15T11:25:16.260233+0000","last_clean":"2024-01-15T11:25:16.260233+0000","last_became_active":"2024-01-15T11:25:16.254438+0000","last_became_peered":"2024-01-15T11:25:16.254438+0000","last_unstale":"2024-01-15T11:25:16.260233+0000","last_undegraded":"2024-01-15T11:25:16.260233+0000","last_fullsized":"2024-01-15T11:25:16.260233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260272+0000","last_change":"2024-01-15T11:25:16.255993+0000","last_active":"2024-01-15T11:25:16.260272+0000","last_peered":"2024-01-15T11:25:16.260272+0000","last_clean":"2024-01-15T11:25:16.260272+0000","last_became_active":"2024-01-15T11:25:16.255718+0000","last_became_peered":"2024-01-15T11:25:16.255718+0000","last_unstale":"2024-01-15T11:25:16.260272+0000","last_undegraded":"2024-01-15T11:25:16.260272+0000","last_fullsized":"2024-01-15T11:25:16.260272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.268918+0000","last_change":"2024-01-15T11:25:16.262620+0000","last_active":"2024-01-15T11:25:16.268918+0000","last_peered":"2024-01-15T11:25:16.268918+0000","last_clean":"2024-01-15T11:25:16.268918+0000","last_became_active":"2024-01-15T11:25:16.262147+0000","last_became_peered":"2024-01-15T11:25:16.262147+0000","last_unstale":"2024-01-15T11:25:16.268918+0000","last_undegraded":"2024-01-15T11:25:16.268918+0000","last_fullsized":"2024-01-15T11:25:16.268918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:26.801860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.266228+0000","last_change":"2024-01-15T11:25:16.252826+0000","last_active":"2024-01-15T11:25:16.266228+0000","last_peered":"2024-01-15T11:25:16.266228+0000","last_clean":"2024-01-15T11:25:16.266228+0000","last_became_active":"2024-01-15T11:25:16.251976+0000","last_became_peered":"2024-01-15T11:25:16.251976+0000","last_unstale":"2024-01-15T11:25:16.266228+0000","last_undegraded":"2024-01-15T11:25:16.266228+0000","last_fullsized":"2024-01-15T11:25:16.266228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:17:27.965755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.273315+0000","last_change":"2024-01-15T11:25:16.266650+0000","last_active":"2024-01-15T11:25:16.273315+0000","last_peered":"2024-01-15T11:25:16.273315+0000","last_clean":"2024-01-15T11:25:16.273315+0000","last_became_active":"2024-01-15T11:25:16.266157+0000","last_became_peered":"2024-01-15T11:25:16.266157+0000","last_unstale":"2024-01-15T11:25:16.273315+0000","last_undegraded":"2024-01-15T11:25:16.273315+0000","last_fullsized":"2024-01-15T11:25:16.273315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:56:33.010927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.266725+0000","last_change":"2024-01-15T11:25:16.255535+0000","last_active":"2024-01-15T11:25:16.266725+0000","last_peered":"2024-01-15T11:25:16.266725+0000","last_clean":"2024-01-15T11:25:16.266725+0000","last_became_active":"2024-01-15T11:25:16.255248+0000","last_became_peered":"2024-01-15T11:25:16.255248+0000","last_unstale":"2024-01-15T11:25:16.266725+0000","last_undegraded":"2024-01-15T11:25:16.266725+0000","last_fullsized":"2024-01-15T11:25:16.266725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:26.586428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260526+0000","last_change":"2024-01-15T11:25:16.258135+0000","last_active":"2024-01-15T11:25:16.260526+0000","last_peered":"2024-01-15T11:25:16.260526+0000","last_clean":"2024-01-15T11:25:16.260526+0000","last_became_active":"2024-01-15T11:25:16.257859+0000","last_became_peered":"2024-01-15T11:25:16.257859+0000","last_unstale":"2024-01-15T11:25:16.260526+0000","last_undegraded":"2024-01-15T11:25:16.260526+0000","last_fullsized":"2024-01-15T11:25:16.260526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:26.386576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.270443+0000","last_change":"2024-01-15T11:25:16.261234+0000","last_active":"2024-01-15T11:25:16.270443+0000","last_peered":"2024-01-15T11:25:16.270443+0000","last_clean":"2024-01-15T11:25:16.270443+0000","last_became_active":"2024-01-15T11:25:16.260952+0000","last_became_peered":"2024-01-15T11:25:16.260952+0000","last_unstale":"2024-01-15T11:25:16.270443+0000","last_undegraded":"2024-01-15T11:25:16.270443+0000","last_fullsized":"2024-01-15T11:25:16.270443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:15:35.970176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272545+0000","last_change":"2024-01-15T11:25:16.266369+0000","last_active":"2024-01-15T11:25:16.272545+0000","last_peered":"2024-01-15T11:25:16.272545+0000","last_clean":"2024-01-15T11:25:16.272545+0000","last_became_active":"2024-01-15T11:25:16.266091+0000","last_became_peered":"2024-01-15T11:25:16.266091+0000","last_unstale":"2024-01-15T11:25:16.272545+0000","last_undegraded":"2024-01-15T11:25:16.272545+0000","last_fullsized":"2024-01-15T11:25:16.272545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:05.228308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272409+0000","last_change":"2024-01-15T11:25:16.259582+0000","last_active":"2024-01-15T11:25:16.272409+0000","last_peered":"2024-01-15T11:25:16.272409+0000","last_clean":"2024-01-15T11:25:16.272409+0000","last_became_active":"2024-01-15T11:25:16.259270+0000","last_became_peered":"2024-01-15T11:25:16.259270+0000","last_unstale":"2024-01-15T11:25:16.272409+0000","last_undegraded":"2024-01-15T11:25:16.272409+0000","last_fullsized":"2024-01-15T11:25:16.272409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275816+0000","last_change":"2024-01-15T11:25:16.274052+0000","last_active":"2024-01-15T11:25:16.275816+0000","last_peered":"2024-01-15T11:25:16.275816+0000","last_clean":"2024-01-15T11:25:16.275816+0000","last_became_active":"2024-01-15T11:25:16.273798+0000","last_became_peered":"2024-01-15T11:25:16.273798+0000","last_unstale":"2024-01-15T11:25:16.275816+0000","last_undegraded":"2024-01-15T11:25:16.275816+0000","last_fullsized":"2024-01-15T11:25:16.275816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:19:46.378784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.253156+0000","last_change":"2024-01-15T11:25:16.250458+0000","last_active":"2024-01-15T11:25:16.253156+0000","last_peered":"2024-01-15T11:25:16.253156+0000","last_clean":"2024-01-15T11:25:16.253156+0000","last_became_active":"2024-01-15T11:25:16.250262+0000","last_became_peered":"2024-01-15T11:25:16.250262+0000","last_unstale":"2024-01-15T11:25:16.253156+0000","last_undegraded":"2024-01-15T11:25:16.253156+0000","last_fullsized":"2024-01-15T11:25:16.253156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:33.426251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.258679+0000","last_change":"2024-01-15T11:25:16.253950+0000","last_active":"2024-01-15T11:25:16.258679+0000","last_peered":"2024-01-15T11:25:16.258679+0000","last_clean":"2024-01-15T11:25:16.258679+0000","last_became_active":"2024-01-15T11:25:16.253667+0000","last_became_peered":"2024-01-15T11:25:16.253667+0000","last_unstale":"2024-01-15T11:25:16.258679+0000","last_undegraded":"2024-01-15T11:25:16.258679+0000","last_fullsized":"2024-01-15T11:25:16.258679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:13.140323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.269756+0000","last_change":"2024-01-15T11:25:16.268387+0000","last_active":"2024-01-15T11:25:16.269756+0000","last_peered":"2024-01-15T11:25:16.269756+0000","last_clean":"2024-01-15T11:25:16.269756+0000","last_became_active":"2024-01-15T11:25:16.268088+0000","last_became_peered":"2024-01-15T11:25:16.268088+0000","last_unstale":"2024-01-15T11:25:16.269756+0000","last_undegraded":"2024-01-15T11:25:16.269756+0000","last_fullsized":"2024-01-15T11:25:16.269756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:59.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.266691+0000","last_change":"2024-01-15T11:25:16.261989+0000","last_active":"2024-01-15T11:25:16.266691+0000","last_peered":"2024-01-15T11:25:16.266691+0000","last_clean":"2024-01-15T11:25:16.266691+0000","last_became_active":"2024-01-15T11:25:16.261260+0000","last_became_peered":"2024-01-15T11:25:16.261260+0000","last_unstale":"2024-01-15T11:25:16.266691+0000","last_undegraded":"2024-01-15T11:25:16.266691+0000","last_fullsized":"2024-01-15T11:25:16.266691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.274834+0000","last_change":"2024-01-15T11:25:16.270904+0000","last_active":"2024-01-15T11:25:16.274834+0000","last_peered":"2024-01-15T11:25:16.274834+0000","last_clean":"2024-01-15T11:25:16.274834+0000","last_became_active":"2024-01-15T11:25:16.270611+0000","last_became_peered":"2024-01-15T11:25:16.270611+0000","last_unstale":"2024-01-15T11:25:16.274834+0000","last_undegraded":"2024-01-15T11:25:16.274834+0000","last_fullsized":"2024-01-15T11:25:16.274834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:54.759984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.261115+0000","last_change":"2024-01-15T11:25:16.257019+0000","last_active":"2024-01-15T11:25:16.261115+0000","last_peered":"2024-01-15T11:25:16.261115+0000","last_clean":"2024-01-15T11:25:16.261115+0000","last_became_active":"2024-01-15T11:25:16.256278+0000","last_became_peered":"2024-01-15T11:25:16.256278+0000","last_unstale":"2024-01-15T11:25:16.261115+0000","last_undegraded":"2024-01-15T11:25:16.261115+0000","last_fullsized":"2024-01-15T11:25:16.261115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.251622+0000","last_change":"2024-01-15T11:25:16.248798+0000","last_active":"2024-01-15T11:25:16.251622+0000","last_peered":"2024-01-15T11:25:16.251622+0000","last_clean":"2024-01-15T11:25:16.251622+0000","last_became_active":"2024-01-15T11:25:16.248488+0000","last_became_peered":"2024-01-15T11:25:16.248488+0000","last_unstale":"2024-01-15T11:25:16.251622+0000","last_undegraded":"2024-01-15T11:25:16.251622+0000","last_fullsized":"2024-01-15T11:25:16.251622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:26.386576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.276047+0000","last_change":"2024-01-15T11:25:15.233384+0000","last_active":"2024-01-15T11:25:16.276047+0000","last_peered":"2024-01-15T11:25:16.276047+0000","last_clean":"2024-01-15T11:25:16.276047+0000","last_became_active":"2024-01-15T11:25:15.233134+0000","last_became_peered":"2024-01-15T11:25:15.233134+0000","last_unstale":"2024-01-15T11:25:16.276047+0000","last_undegraded":"2024-01-15T11:25:16.276047+0000","last_fullsized":"2024-01-15T11:25:16.276047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260826+0000","last_change":"2024-01-15T11:25:16.256579+0000","last_active":"2024-01-15T11:25:16.260826+0000","last_peered":"2024-01-15T11:25:16.260826+0000","last_clean":"2024-01-15T11:25:16.260826+0000","last_became_active":"2024-01-15T11:25:16.256320+0000","last_became_peered":"2024-01-15T11:25:16.256320+0000","last_unstale":"2024-01-15T11:25:16.260826+0000","last_undegraded":"2024-01-15T11:25:16.260826+0000","last_fullsized":"2024-01-15T11:25:16.260826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.271304+0000","last_change":"2024-01-15T11:25:15.230417+0000","last_active":"2024-01-15T11:25:16.271304+0000","last_peered":"2024-01-15T11:25:16.271304+0000","last_clean":"2024-01-15T11:25:16.271304+0000","last_became_active":"2024-01-15T11:25:15.230135+0000","last_became_peered":"2024-01-15T11:25:15.230135+0000","last_unstale":"2024-01-15T11:25:16.271304+0000","last_undegraded":"2024-01-15T11:25:16.271304+0000","last_fullsized":"2024-01-15T11:25:16.271304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:25.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.263184+0000","last_change":"2024-01-15T11:25:16.259985+0000","last_active":"2024-01-15T11:25:16.263184+0000","last_peered":"2024-01-15T11:25:16.263184+0000","last_clean":"2024-01-15T11:25:16.263184+0000","last_became_active":"2024-01-15T11:25:16.259697+0000","last_became_peered":"2024-01-15T11:25:16.259697+0000","last_unstale":"2024-01-15T11:25:16.263184+0000","last_undegraded":"2024-01-15T11:25:16.263184+0000","last_fullsized":"2024-01-15T11:25:16.263184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:26.586428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272821+0000","last_change":"2024-01-15T11:25:15.238695+0000","last_active":"2024-01-15T11:25:16.272821+0000","last_peered":"2024-01-15T11:25:16.272821+0000","last_clean":"2024-01-15T11:25:16.272821+0000","last_became_active":"2024-01-15T11:25:15.238377+0000","last_became_peered":"2024-01-15T11:25:15.238377+0000","last_unstale":"2024-01-15T11:25:16.272821+0000","last_undegraded":"2024-01-15T11:25:16.272821+0000","last_fullsized":"2024-01-15T11:25:16.272821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:48.780156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.268942+0000","last_change":"2024-01-15T11:25:16.264379+0000","last_active":"2024-01-15T11:25:16.268942+0000","last_peered":"2024-01-15T11:25:16.268942+0000","last_clean":"2024-01-15T11:25:16.268942+0000","last_became_active":"2024-01-15T11:25:16.264121+0000","last_became_peered":"2024-01-15T11:25:16.264121+0000","last_unstale":"2024-01-15T11:25:16.268942+0000","last_undegraded":"2024-01-15T11:25:16.268942+0000","last_fullsized":"2024-01-15T11:25:16.268942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:17.504194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275466+0000","last_change":"2024-01-15T11:25:15.237423+0000","last_active":"2024-01-15T11:25:16.275466+0000","last_peered":"2024-01-15T11:25:16.275466+0000","last_clean":"2024-01-15T11:25:16.275466+0000","last_became_active":"2024-01-15T11:25:15.237121+0000","last_became_peered":"2024-01-15T11:25:15.237121+0000","last_unstale":"2024-01-15T11:25:16.275466+0000","last_undegraded":"2024-01-15T11:25:16.275466+0000","last_fullsized":"2024-01-15T11:25:16.275466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:04.223510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.273258+0000","last_change":"2024-01-15T11:25:16.266842+0000","last_active":"2024-01-15T11:25:16.273258+0000","last_peered":"2024-01-15T11:25:16.273258+0000","last_clean":"2024-01-15T11:25:16.273258+0000","last_became_active":"2024-01-15T11:25:16.266606+0000","last_became_peered":"2024-01-15T11:25:16.266606+0000","last_unstale":"2024-01-15T11:25:16.273258+0000","last_undegraded":"2024-01-15T11:25:16.273258+0000","last_fullsized":"2024-01-15T11:25:16.273258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275555+0000","last_change":"2024-01-15T11:25:15.232263+0000","last_active":"2024-01-15T11:25:16.275555+0000","last_peered":"2024-01-15T11:25:16.275555+0000","last_clean":"2024-01-15T11:25:16.275555+0000","last_became_active":"2024-01-15T11:25:15.231976+0000","last_became_peered":"2024-01-15T11:25:15.231976+0000","last_unstale":"2024-01-15T11:25:16.275555+0000","last_undegraded":"2024-01-15T11:25:16.275555+0000","last_fullsized":"2024-01-15T11:25:16.275555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:25.797063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.268081+0000","last_change":"2024-01-15T11:25:16.263430+0000","last_active":"2024-01-15T11:25:16.268081+0000","last_peered":"2024-01-15T11:25:16.268081+0000","last_clean":"2024-01-15T11:25:16.268081+0000","last_became_active":"2024-01-15T11:25:16.261944+0000","last_became_peered":"2024-01-15T11:25:16.261944+0000","last_unstale":"2024-01-15T11:25:16.268081+0000","last_undegraded":"2024-01-15T11:25:16.268081+0000","last_fullsized":"2024-01-15T11:25:16.268081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:13.140323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275875+0000","last_change":"2024-01-15T11:25:15.235451+0000","last_active":"2024-01-15T11:25:16.275875+0000","last_peered":"2024-01-15T11:25:16.275875+0000","last_clean":"2024-01-15T11:25:16.275875+0000","last_became_active":"2024-01-15T11:25:15.235166+0000","last_became_peered":"2024-01-15T11:25:15.235166+0000","last_unstale":"2024-01-15T11:25:16.275875+0000","last_undegraded":"2024-01-15T11:25:16.275875+0000","last_fullsized":"2024-01-15T11:25:16.275875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.263710+0000","last_change":"2024-01-15T11:25:16.262932+0000","last_active":"2024-01-15T11:25:16.263710+0000","last_peered":"2024-01-15T11:25:16.263710+0000","last_clean":"2024-01-15T11:25:16.263710+0000","last_became_active":"2024-01-15T11:25:16.262667+0000","last_became_peered":"2024-01-15T11:25:16.262667+0000","last_unstale":"2024-01-15T11:25:16.263710+0000","last_undegraded":"2024-01-15T11:25:16.263710+0000","last_fullsized":"2024-01-15T11:25:16.263710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:19:46.378784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.244597+0000","last_change":"2024-01-15T11:25:15.232286+0000","last_active":"2024-01-15T11:25:16.244597+0000","last_peered":"2024-01-15T11:25:16.244597+0000","last_clean":"2024-01-15T11:25:16.244597+0000","last_became_active":"2024-01-15T11:25:15.230764+0000","last_became_peered":"2024-01-15T11:25:15.230764+0000","last_unstale":"2024-01-15T11:25:16.244597+0000","last_undegraded":"2024-01-15T11:25:16.244597+0000","last_fullsized":"2024-01-15T11:25:16.244597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:53.755186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260488+0000","last_change":"2024-01-15T11:25:16.254427+0000","last_active":"2024-01-15T11:25:16.260488+0000","last_peered":"2024-01-15T11:25:16.260488+0000","last_clean":"2024-01-15T11:25:16.260488+0000","last_became_active":"2024-01-15T11:25:16.253271+0000","last_became_peered":"2024-01-15T11:25:16.253271+0000","last_unstale":"2024-01-15T11:25:16.260488+0000","last_undegraded":"2024-01-15T11:25:16.260488+0000","last_fullsized":"2024-01-15T11:25:16.260488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.431909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.250014+0000","last_change":"2024-01-15T11:25:15.231253+0000","last_active":"2024-01-15T11:25:16.250014+0000","last_peered":"2024-01-15T11:25:16.250014+0000","last_clean":"2024-01-15T11:25:16.250014+0000","last_became_active":"2024-01-15T11:25:15.230935+0000","last_became_peered":"2024-01-15T11:25:15.230935+0000","last_unstale":"2024-01-15T11:25:16.250014+0000","last_undegraded":"2024-01-15T11:25:16.250014+0000","last_fullsized":"2024-01-15T11:25:16.250014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:04.223510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260112+0000","last_change":"2024-01-15T11:25:16.256798+0000","last_active":"2024-01-15T11:25:16.260112+0000","last_peered":"2024-01-15T11:25:16.260112+0000","last_clean":"2024-01-15T11:25:16.260112+0000","last_became_active":"2024-01-15T11:25:16.256540+0000","last_became_peered":"2024-01-15T11:25:16.256540+0000","last_unstale":"2024-01-15T11:25:16.260112+0000","last_undegraded":"2024-01-15T11:25:16.260112+0000","last_fullsized":"2024-01-15T11:25:16.260112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.431909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.251444+0000","last_change":"2024-01-15T11:25:15.236440+0000","last_active":"2024-01-15T11:25:16.251444+0000","last_peered":"2024-01-15T11:25:16.251444+0000","last_clean":"2024-01-15T11:25:16.251444+0000","last_became_active":"2024-01-15T11:25:15.236034+0000","last_became_peered":"2024-01-15T11:25:15.236034+0000","last_unstale":"2024-01-15T11:25:16.251444+0000","last_undegraded":"2024-01-15T11:25:16.251444+0000","last_fullsized":"2024-01-15T11:25:16.251444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:25.797063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.274623+0000","last_change":"2024-01-15T11:25:16.270509+0000","last_active":"2024-01-15T11:25:16.274623+0000","last_peered":"2024-01-15T11:25:16.274623+0000","last_clean":"2024-01-15T11:25:16.274623+0000","last_became_active":"2024-01-15T11:25:16.270319+0000","last_became_peered":"2024-01-15T11:25:16.270319+0000","last_unstale":"2024-01-15T11:25:16.274623+0000","last_undegraded":"2024-01-15T11:25:16.274623+0000","last_fullsized":"2024-01-15T11:25:16.274623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:59.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.252880+0000","last_change":"2024-01-15T11:25:15.235314+0000","last_active":"2024-01-15T11:25:16.252880+0000","last_peered":"2024-01-15T11:25:16.252880+0000","last_clean":"2024-01-15T11:25:16.252880+0000","last_became_active":"2024-01-15T11:25:15.232515+0000","last_became_peered":"2024-01-15T11:25:15.232515+0000","last_unstale":"2024-01-15T11:25:16.252880+0000","last_undegraded":"2024-01-15T11:25:16.252880+0000","last_fullsized":"2024-01-15T11:25:16.252880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:32.421454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.263204+0000","last_change":"2024-01-15T11:25:16.260223+0000","last_active":"2024-01-15T11:25:16.263204+0000","last_peered":"2024-01-15T11:25:16.263204+0000","last_clean":"2024-01-15T11:25:16.263204+0000","last_became_active":"2024-01-15T11:25:16.259365+0000","last_became_peered":"2024-01-15T11:25:16.259365+0000","last_unstale":"2024-01-15T11:25:16.263204+0000","last_undegraded":"2024-01-15T11:25:16.263204+0000","last_fullsized":"2024-01-15T11:25:16.263204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.273490+0000","last_change":"2024-01-15T11:25:15.233433+0000","last_active":"2024-01-15T11:25:16.273490+0000","last_peered":"2024-01-15T11:25:16.273490+0000","last_clean":"2024-01-15T11:25:16.273490+0000","last_became_active":"2024-01-15T11:25:15.233046+0000","last_became_peered":"2024-01-15T11:25:15.233046+0000","last_unstale":"2024-01-15T11:25:16.273490+0000","last_undegraded":"2024-01-15T11:25:16.273490+0000","last_fullsized":"2024-01-15T11:25:16.273490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:25.797063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.277312+0000","last_change":"2024-01-15T11:25:16.275023+0000","last_active":"2024-01-15T11:25:16.277312+0000","last_peered":"2024-01-15T11:25:16.277312+0000","last_clean":"2024-01-15T11:25:16.277312+0000","last_became_active":"2024-01-15T11:25:16.274744+0000","last_became_peered":"2024-01-15T11:25:16.274744+0000","last_unstale":"2024-01-15T11:25:16.277312+0000","last_undegraded":"2024-01-15T11:25:16.277312+0000","last_fullsized":"2024-01-15T11:25:16.277312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:43.332244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.277325+0000","last_change":"2024-01-15T11:25:15.235460+0000","last_active":"2024-01-15T11:25:16.277325+0000","last_peered":"2024-01-15T11:25:16.277325+0000","last_clean":"2024-01-15T11:25:16.277325+0000","last_became_active":"2024-01-15T11:25:15.235194+0000","last_became_peered":"2024-01-15T11:25:15.235194+0000","last_unstale":"2024-01-15T11:25:16.277325+0000","last_undegraded":"2024-01-15T11:25:16.277325+0000","last_fullsized":"2024-01-15T11:25:16.277325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:32.421454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.266744+0000","last_change":"2024-01-15T11:25:16.254349+0000","last_active":"2024-01-15T11:25:16.266744+0000","last_peered":"2024-01-15T11:25:16.266744+0000","last_clean":"2024-01-15T11:25:16.266744+0000","last_became_active":"2024-01-15T11:25:16.254068+0000","last_became_peered":"2024-01-15T11:25:16.254068+0000","last_unstale":"2024-01-15T11:25:16.266744+0000","last_undegraded":"2024-01-15T11:25:16.266744+0000","last_fullsized":"2024-01-15T11:25:16.266744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:56:33.010927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275958+0000","last_change":"2024-01-15T11:25:15.236870+0000","last_active":"2024-01-15T11:25:16.275958+0000","last_peered":"2024-01-15T11:25:16.275958+0000","last_clean":"2024-01-15T11:25:16.275958+0000","last_became_active":"2024-01-15T11:25:15.236597+0000","last_became_peered":"2024-01-15T11:25:15.236597+0000","last_unstale":"2024-01-15T11:25:16.275958+0000","last_undegraded":"2024-01-15T11:25:16.275958+0000","last_fullsized":"2024-01-15T11:25:16.275958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:04.223510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260775+0000","last_change":"2024-01-15T11:25:16.257709+0000","last_active":"2024-01-15T11:25:16.260775+0000","last_peered":"2024-01-15T11:25:16.260775+0000","last_clean":"2024-01-15T11:25:16.260775+0000","last_became_active":"2024-01-15T11:25:16.257415+0000","last_became_peered":"2024-01-15T11:25:16.257415+0000","last_unstale":"2024-01-15T11:25:16.260775+0000","last_undegraded":"2024-01-15T11:25:16.260775+0000","last_fullsized":"2024-01-15T11:25:16.260775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:43.332244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.259168+0000","last_change":"2024-01-15T11:25:15.233904+0000","last_active":"2024-01-15T11:25:16.259168+0000","last_peered":"2024-01-15T11:25:16.259168+0000","last_clean":"2024-01-15T11:25:16.259168+0000","last_became_active":"2024-01-15T11:25:15.232303+0000","last_became_peered":"2024-01-15T11:25:15.232303+0000","last_unstale":"2024-01-15T11:25:16.259168+0000","last_undegraded":"2024-01-15T11:25:16.259168+0000","last_fullsized":"2024-01-15T11:25:16.259168+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:14.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.271334+0000","last_change":"2024-01-15T11:25:16.265440+0000","last_active":"2024-01-15T11:25:16.271334+0000","last_peered":"2024-01-15T11:25:16.271334+0000","last_clean":"2024-01-15T11:25:16.271334+0000","last_became_active":"2024-01-15T11:25:16.265124+0000","last_became_peered":"2024-01-15T11:25:16.265124+0000","last_unstale":"2024-01-15T11:25:16.271334+0000","last_undegraded":"2024-01-15T11:25:16.271334+0000","last_fullsized":"2024-01-15T11:25:16.271334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:54.759984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.249605+0000","last_change":"2024-01-15T11:25:15.230668+0000","last_active":"2024-01-15T11:25:16.249605+0000","last_peered":"2024-01-15T11:25:16.249605+0000","last_clean":"2024-01-15T11:25:16.249605+0000","last_became_active":"2024-01-15T11:25:15.230323+0000","last_became_peered":"2024-01-15T11:25:15.230323+0000","last_unstale":"2024-01-15T11:25:16.249605+0000","last_undegraded":"2024-01-15T11:25:16.249605+0000","last_fullsized":"2024-01-15T11:25:16.249605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:25.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272584+0000","last_change":"2024-01-15T11:25:16.267799+0000","last_active":"2024-01-15T11:25:16.272584+0000","last_peered":"2024-01-15T11:25:16.272584+0000","last_clean":"2024-01-15T11:25:16.272584+0000","last_became_active":"2024-01-15T11:25:16.266612+0000","last_became_peered":"2024-01-15T11:25:16.266612+0000","last_unstale":"2024-01-15T11:25:16.272584+0000","last_undegraded":"2024-01-15T11:25:16.272584+0000","last_fullsized":"2024-01-15T11:25:16.272584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:17.504194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272359+0000","last_change":"2024-01-15T11:25:15.236904+0000","last_active":"2024-01-15T11:25:16.272359+0000","last_peered":"2024-01-15T11:25:16.272359+0000","last_clean":"2024-01-15T11:25:16.272359+0000","last_became_active":"2024-01-15T11:25:15.236628+0000","last_became_peered":"2024-01-15T11:25:15.236628+0000","last_unstale":"2024-01-15T11:25:16.272359+0000","last_undegraded":"2024-01-15T11:25:16.272359+0000","last_fullsized":"2024-01-15T11:25:16.272359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:42.327446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275356+0000","last_change":"2024-01-15T11:25:16.271784+0000","last_active":"2024-01-15T11:25:16.275356+0000","last_peered":"2024-01-15T11:25:16.275356+0000","last_clean":"2024-01-15T11:25:16.275356+0000","last_became_active":"2024-01-15T11:25:16.271432+0000","last_became_peered":"2024-01-15T11:25:16.271432+0000","last_unstale":"2024-01-15T11:25:16.275356+0000","last_undegraded":"2024-01-15T11:25:16.275356+0000","last_fullsized":"2024-01-15T11:25:16.275356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:33.426251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275216+0000","last_change":"2024-01-15T11:25:15.238136+0000","last_active":"2024-01-15T11:25:16.275216+0000","last_peered":"2024-01-15T11:25:16.275216+0000","last_clean":"2024-01-15T11:25:16.275216+0000","last_became_active":"2024-01-15T11:25:15.237522+0000","last_became_peered":"2024-01-15T11:25:15.237522+0000","last_unstale":"2024-01-15T11:25:16.275216+0000","last_undegraded":"2024-01-15T11:25:16.275216+0000","last_fullsized":"2024-01-15T11:25:16.275216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:16.499396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.270209+0000","last_change":"2024-01-15T11:25:16.263503+0000","last_active":"2024-01-15T11:25:16.270209+0000","last_peered":"2024-01-15T11:25:16.270209+0000","last_clean":"2024-01-15T11:25:16.270209+0000","last_became_active":"2024-01-15T11:25:16.263208+0000","last_became_peered":"2024-01-15T11:25:16.263208+0000","last_unstale":"2024-01-15T11:25:16.270209+0000","last_undegraded":"2024-01-15T11:25:16.270209+0000","last_fullsized":"2024-01-15T11:25:16.270209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:26.801860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.248350+0000","last_change":"2024-01-15T11:25:15.229209+0000","last_active":"2024-01-15T11:25:16.248350+0000","last_peered":"2024-01-15T11:25:16.248350+0000","last_clean":"2024-01-15T11:25:16.248350+0000","last_became_active":"2024-01-15T11:25:15.228899+0000","last_became_peered":"2024-01-15T11:25:15.228899+0000","last_unstale":"2024-01-15T11:25:16.248350+0000","last_undegraded":"2024-01-15T11:25:16.248350+0000","last_fullsized":"2024-01-15T11:25:16.248350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:56:32.006130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.266818+0000","last_change":"2024-01-15T11:25:16.259973+0000","last_active":"2024-01-15T11:25:16.266818+0000","last_peered":"2024-01-15T11:25:16.266818+0000","last_clean":"2024-01-15T11:25:16.266818+0000","last_became_active":"2024-01-15T11:25:16.259589+0000","last_became_peered":"2024-01-15T11:25:16.259589+0000","last_unstale":"2024-01-15T11:25:16.266818+0000","last_undegraded":"2024-01-15T11:25:16.266818+0000","last_fullsized":"2024-01-15T11:25:16.266818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.431909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.258986+0000","last_change":"2024-01-15T11:25:15.234071+0000","last_active":"2024-01-15T11:25:16.258986+0000","last_peered":"2024-01-15T11:25:16.258986+0000","last_clean":"2024-01-15T11:25:16.258986+0000","last_became_active":"2024-01-15T11:25:15.233460+0000","last_became_peered":"2024-01-15T11:25:15.233460+0000","last_unstale":"2024-01-15T11:25:16.258986+0000","last_undegraded":"2024-01-15T11:25:16.258986+0000","last_fullsized":"2024-01-15T11:25:16.258986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:30.739157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275800+0000","last_change":"2024-01-15T11:25:16.268001+0000","last_active":"2024-01-15T11:25:16.275800+0000","last_peered":"2024-01-15T11:25:16.275800+0000","last_clean":"2024-01-15T11:25:16.275800+0000","last_became_active":"2024-01-15T11:25:16.267277+0000","last_became_peered":"2024-01-15T11:25:16.267277+0000","last_unstale":"2024-01-15T11:25:16.275800+0000","last_undegraded":"2024-01-15T11:25:16.275800+0000","last_fullsized":"2024-01-15T11:25:16.275800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:59.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260263+0000","last_change":"2024-01-15T11:25:16.258019+0000","last_active":"2024-01-15T11:25:16.260263+0000","last_peered":"2024-01-15T11:25:16.260263+0000","last_clean":"2024-01-15T11:25:16.260263+0000","last_became_active":"2024-01-15T11:25:16.257755+0000","last_became_peered":"2024-01-15T11:25:16.257755+0000","last_unstale":"2024-01-15T11:25:16.260263+0000","last_undegraded":"2024-01-15T11:25:16.260263+0000","last_fullsized":"2024-01-15T11:25:16.260263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:05.228308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.243043+0000","last_change":"2024-01-15T11:25:15.235627+0000","last_active":"2024-01-15T11:25:16.243043+0000","last_peered":"2024-01-15T11:25:16.243043+0000","last_clean":"2024-01-15T11:25:16.243043+0000","last_became_active":"2024-01-15T11:25:15.235308+0000","last_became_peered":"2024-01-15T11:25:15.235308+0000","last_unstale":"2024-01-15T11:25:16.243043+0000","last_undegraded":"2024-01-15T11:25:16.243043+0000","last_fullsized":"2024-01-15T11:25:16.243043+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:58.480006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.261119+0000","last_change":"2024-01-15T11:25:16.260147+0000","last_active":"2024-01-15T11:25:16.261119+0000","last_peered":"2024-01-15T11:25:16.261119+0000","last_clean":"2024-01-15T11:25:16.261119+0000","last_became_active":"2024-01-15T11:25:16.259836+0000","last_became_peered":"2024-01-15T11:25:16.259836+0000","last_unstale":"2024-01-15T11:25:16.261119+0000","last_undegraded":"2024-01-15T11:25:16.261119+0000","last_fullsized":"2024-01-15T11:25:16.261119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:59.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.250071+0000","last_change":"2024-01-15T11:25:15.233565+0000","last_active":"2024-01-15T11:25:16.250071+0000","last_peered":"2024-01-15T11:25:16.250071+0000","last_clean":"2024-01-15T11:25:16.250071+0000","last_became_active":"2024-01-15T11:25:15.231919+0000","last_became_peered":"2024-01-15T11:25:15.231919+0000","last_unstale":"2024-01-15T11:25:16.250071+0000","last_undegraded":"2024-01-15T11:25:16.250071+0000","last_fullsized":"2024-01-15T11:25:16.250071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-15T11:25:15.227419+0000","last_change":"2024-01-15T11:25:03.506154+0000","last_active":"2024-01-15T11:25:15.227419+0000","last_peered":"2024-01-15T11:25:15.227419+0000","last_clean":"2024-01-15T11:25:15.227419+0000","last_became_active":"2024-01-15T11:25:03.504996+0000","last_became_peered":"2024-01-15T11:25:03.504996+0000","last_unstale":"2024-01-15T11:25:15.227419+0000","last_undegraded":"2024-01-15T11:25:15.227419+0000","last_fullsized":"2024-01-15T11:25:15.227419+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:02.494136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:02.494136+0000","last_clean_scrub_stamp":"2024-01-15T11:25:02.494136+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:22:33.043449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.271694+0000","last_change":"2024-01-15T11:25:16.270301+0000","last_active":"2024-01-15T11:25:16.271694+0000","last_peered":"2024-01-15T11:25:16.271694+0000","last_clean":"2024-01-15T11:25:16.271694+0000","last_became_active":"2024-01-15T11:25:16.269969+0000","last_became_peered":"2024-01-15T11:25:16.269969+0000","last_unstale":"2024-01-15T11:25:16.271694+0000","last_undegraded":"2024-01-15T11:25:16.271694+0000","last_fullsized":"2024-01-15T11:25:16.271694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.243429+0000","last_change":"2024-01-15T11:25:15.233382+0000","last_active":"2024-01-15T11:25:16.243429+0000","last_peered":"2024-01-15T11:25:16.243429+0000","last_clean":"2024-01-15T11:25:16.243429+0000","last_became_active":"2024-01-15T11:25:15.233111+0000","last_became_peered":"2024-01-15T11:25:15.233111+0000","last_unstale":"2024-01-15T11:25:16.243429+0000","last_undegraded":"2024-01-15T11:25:16.243429+0000","last_fullsized":"2024-01-15T11:25:16.243429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:50.865963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.244625+0000","last_change":"2024-01-15T11:25:15.237505+0000","last_active":"2024-01-15T11:25:16.244625+0000","last_peered":"2024-01-15T11:25:16.244625+0000","last_clean":"2024-01-15T11:25:16.244625+0000","last_became_active":"2024-01-15T11:25:15.237206+0000","last_became_peered":"2024-01-15T11:25:15.237206+0000","last_unstale":"2024-01-15T11:25:16.244625+0000","last_undegraded":"2024-01-15T11:25:16.244625+0000","last_fullsized":"2024-01-15T11:25:16.244625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:14.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.258881+0000","last_change":"2024-01-15T11:25:16.254087+0000","last_active":"2024-01-15T11:25:16.258881+0000","last_peered":"2024-01-15T11:25:16.258881+0000","last_clean":"2024-01-15T11:25:16.258881+0000","last_became_active":"2024-01-15T11:25:16.253790+0000","last_became_peered":"2024-01-15T11:25:16.253790+0000","last_unstale":"2024-01-15T11:25:16.258881+0000","last_undegraded":"2024-01-15T11:25:16.258881+0000","last_fullsized":"2024-01-15T11:25:16.258881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.247853+0000","last_change":"2024-01-15T11:25:15.240170+0000","last_active":"2024-01-15T11:25:16.247853+0000","last_peered":"2024-01-15T11:25:16.247853+0000","last_clean":"2024-01-15T11:25:16.247853+0000","last_became_active":"2024-01-15T11:25:15.239842+0000","last_became_peered":"2024-01-15T11:25:15.239842+0000","last_unstale":"2024-01-15T11:25:16.247853+0000","last_undegraded":"2024-01-15T11:25:16.247853+0000","last_fullsized":"2024-01-15T11:25:16.247853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:25.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.255085+0000","last_change":"2024-01-15T11:25:16.247896+0000","last_active":"2024-01-15T11:25:16.255085+0000","last_peered":"2024-01-15T11:25:16.255085+0000","last_clean":"2024-01-15T11:25:16.255085+0000","last_became_active":"2024-01-15T11:25:16.247606+0000","last_became_peered":"2024-01-15T11:25:16.247606+0000","last_unstale":"2024-01-15T11:25:16.255085+0000","last_undegraded":"2024-01-15T11:25:16.255085+0000","last_fullsized":"2024-01-15T11:25:16.255085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272739+0000","last_change":"2024-01-15T11:25:15.233571+0000","last_active":"2024-01-15T11:25:16.272739+0000","last_peered":"2024-01-15T11:25:16.272739+0000","last_clean":"2024-01-15T11:25:16.272739+0000","last_became_active":"2024-01-15T11:25:15.233180+0000","last_became_peered":"2024-01-15T11:25:15.233180+0000","last_unstale":"2024-01-15T11:25:16.272739+0000","last_undegraded":"2024-01-15T11:25:16.272739+0000","last_fullsized":"2024-01-15T11:25:16.272739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:53.755186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.258833+0000","last_change":"2024-01-15T11:25:16.253976+0000","last_active":"2024-01-15T11:25:16.258833+0000","last_peered":"2024-01-15T11:25:16.258833+0000","last_clean":"2024-01-15T11:25:16.258833+0000","last_became_active":"2024-01-15T11:25:16.253600+0000","last_became_peered":"2024-01-15T11:25:16.253600+0000","last_unstale":"2024-01-15T11:25:16.258833+0000","last_undegraded":"2024-01-15T11:25:16.258833+0000","last_fullsized":"2024-01-15T11:25:16.258833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:17.504194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.243339+0000","last_change":"2024-01-15T11:25:15.235253+0000","last_active":"2024-01-15T11:25:16.243339+0000","last_peered":"2024-01-15T11:25:16.243339+0000","last_clean":"2024-01-15T11:25:16.243339+0000","last_became_active":"2024-01-15T11:25:15.234844+0000","last_became_peered":"2024-01-15T11:25:15.234844+0000","last_unstale":"2024-01-15T11:25:16.243339+0000","last_undegraded":"2024-01-15T11:25:16.243339+0000","last_fullsized":"2024-01-15T11:25:16.243339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:18.628480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.264574+0000","last_change":"2024-01-15T11:25:16.257934+0000","last_active":"2024-01-15T11:25:16.264574+0000","last_peered":"2024-01-15T11:25:16.264574+0000","last_clean":"2024-01-15T11:25:16.264574+0000","last_became_active":"2024-01-15T11:25:16.257566+0000","last_became_peered":"2024-01-15T11:25:16.257566+0000","last_unstale":"2024-01-15T11:25:16.264574+0000","last_undegraded":"2024-01-15T11:25:16.264574+0000","last_fullsized":"2024-01-15T11:25:16.264574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:23.083850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.249457+0000","last_change":"2024-01-15T11:25:15.237581+0000","last_active":"2024-01-15T11:25:16.249457+0000","last_peered":"2024-01-15T11:25:16.249457+0000","last_clean":"2024-01-15T11:25:16.249457+0000","last_became_active":"2024-01-15T11:25:15.237319+0000","last_became_peered":"2024-01-15T11:25:15.237319+0000","last_unstale":"2024-01-15T11:25:16.249457+0000","last_undegraded":"2024-01-15T11:25:16.249457+0000","last_fullsized":"2024-01-15T11:25:16.249457+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260508+0000","last_change":"2024-01-15T11:25:16.256476+0000","last_active":"2024-01-15T11:25:16.260508+0000","last_peered":"2024-01-15T11:25:16.260508+0000","last_clean":"2024-01-15T11:25:16.260508+0000","last_became_active":"2024-01-15T11:25:16.256198+0000","last_became_peered":"2024-01-15T11:25:16.256198+0000","last_unstale":"2024-01-15T11:25:16.260508+0000","last_undegraded":"2024-01-15T11:25:16.260508+0000","last_fullsized":"2024-01-15T11:25:16.260508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.248235+0000","last_change":"2024-01-15T11:25:15.235154+0000","last_active":"2024-01-15T11:25:16.248235+0000","last_peered":"2024-01-15T11:25:16.248235+0000","last_clean":"2024-01-15T11:25:16.248235+0000","last_became_active":"2024-01-15T11:25:15.234870+0000","last_became_peered":"2024-01-15T11:25:15.234870+0000","last_unstale":"2024-01-15T11:25:16.248235+0000","last_undegraded":"2024-01-15T11:25:16.248235+0000","last_fullsized":"2024-01-15T11:25:16.248235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:53.755186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260309+0000","last_change":"2024-01-15T11:25:16.254780+0000","last_active":"2024-01-15T11:25:16.260309+0000","last_peered":"2024-01-15T11:25:16.260309+0000","last_clean":"2024-01-15T11:25:16.260309+0000","last_became_active":"2024-01-15T11:25:16.254519+0000","last_became_peered":"2024-01-15T11:25:16.254519+0000","last_unstale":"2024-01-15T11:25:16.260309+0000","last_undegraded":"2024-01-15T11:25:16.260309+0000","last_fullsized":"2024-01-15T11:25:16.260309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:17.504194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.277415+0000","last_change":"2024-01-15T11:25:15.238538+0000","last_active":"2024-01-15T11:25:16.277415+0000","last_peered":"2024-01-15T11:25:16.277415+0000","last_clean":"2024-01-15T11:25:16.277415+0000","last_became_active":"2024-01-15T11:25:15.238222+0000","last_became_peered":"2024-01-15T11:25:15.238222+0000","last_unstale":"2024-01-15T11:25:16.277415+0000","last_undegraded":"2024-01-15T11:25:16.277415+0000","last_fullsized":"2024-01-15T11:25:16.277415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:30.739157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272839+0000","last_change":"2024-01-15T11:25:16.262608+0000","last_active":"2024-01-15T11:25:16.272839+0000","last_peered":"2024-01-15T11:25:16.272839+0000","last_clean":"2024-01-15T11:25:16.272839+0000","last_became_active":"2024-01-15T11:25:16.262318+0000","last_became_peered":"2024-01-15T11:25:16.262318+0000","last_unstale":"2024-01-15T11:25:16.272839+0000","last_undegraded":"2024-01-15T11:25:16.272839+0000","last_fullsized":"2024-01-15T11:25:16.272839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:59.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.242958+0000","last_change":"2024-01-15T11:25:15.232704+0000","last_active":"2024-01-15T11:25:16.242958+0000","last_peered":"2024-01-15T11:25:16.242958+0000","last_clean":"2024-01-15T11:25:16.242958+0000","last_became_active":"2024-01-15T11:25:15.232191+0000","last_became_peered":"2024-01-15T11:25:15.232191+0000","last_unstale":"2024-01-15T11:25:16.242958+0000","last_undegraded":"2024-01-15T11:25:16.242958+0000","last_fullsized":"2024-01-15T11:25:16.242958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:16.499396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.269902+0000","last_change":"2024-01-15T11:25:16.260276+0000","last_active":"2024-01-15T11:25:16.269902+0000","last_peered":"2024-01-15T11:25:16.269902+0000","last_clean":"2024-01-15T11:25:16.269902+0000","last_became_active":"2024-01-15T11:25:16.259962+0000","last_became_peered":"2024-01-15T11:25:16.259962+0000","last_unstale":"2024-01-15T11:25:16.269902+0000","last_undegraded":"2024-01-15T11:25:16.269902+0000","last_fullsized":"2024-01-15T11:25:16.269902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:08:06.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275885+0000","last_change":"2024-01-15T11:25:15.230446+0000","last_active":"2024-01-15T11:25:16.275885+0000","last_peered":"2024-01-15T11:25:16.275885+0000","last_clean":"2024-01-15T11:25:16.275885+0000","last_became_active":"2024-01-15T11:25:15.230176+0000","last_became_peered":"2024-01-15T11:25:15.230176+0000","last_unstale":"2024-01-15T11:25:16.275885+0000","last_undegraded":"2024-01-15T11:25:16.275885+0000","last_fullsized":"2024-01-15T11:25:16.275885+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:32.421454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.268751+0000","last_change":"2024-01-15T11:25:16.262438+0000","last_active":"2024-01-15T11:25:16.268751+0000","last_peered":"2024-01-15T11:25:16.268751+0000","last_clean":"2024-01-15T11:25:16.268751+0000","last_became_active":"2024-01-15T11:25:16.261849+0000","last_became_peered":"2024-01-15T11:25:16.261849+0000","last_unstale":"2024-01-15T11:25:16.268751+0000","last_undegraded":"2024-01-15T11:25:16.268751+0000","last_fullsized":"2024-01-15T11:25:16.268751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:26.586428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.271673+0000","last_change":"2024-01-15T11:25:15.235007+0000","last_active":"2024-01-15T11:25:16.271673+0000","last_peered":"2024-01-15T11:25:16.271673+0000","last_clean":"2024-01-15T11:25:16.271673+0000","last_became_active":"2024-01-15T11:25:15.234709+0000","last_became_peered":"2024-01-15T11:25:15.234709+0000","last_unstale":"2024-01-15T11:25:16.271673+0000","last_undegraded":"2024-01-15T11:25:16.271673+0000","last_fullsized":"2024-01-15T11:25:16.271673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:25.427111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.270455+0000","last_change":"2024-01-15T11:25:16.263407+0000","last_active":"2024-01-15T11:25:16.270455+0000","last_peered":"2024-01-15T11:25:16.270455+0000","last_clean":"2024-01-15T11:25:16.270455+0000","last_became_active":"2024-01-15T11:25:16.263120+0000","last_became_peered":"2024-01-15T11:25:16.263120+0000","last_unstale":"2024-01-15T11:25:16.270455+0000","last_undegraded":"2024-01-15T11:25:16.270455+0000","last_fullsized":"2024-01-15T11:25:16.270455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:28:18.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272390+0000","last_change":"2024-01-15T11:25:16.258910+0000","last_active":"2024-01-15T11:25:16.272390+0000","last_peered":"2024-01-15T11:25:16.272390+0000","last_clean":"2024-01-15T11:25:16.272390+0000","last_became_active":"2024-01-15T11:25:16.258619+0000","last_became_peered":"2024-01-15T11:25:16.258619+0000","last_unstale":"2024-01-15T11:25:16.272390+0000","last_undegraded":"2024-01-15T11:25:16.272390+0000","last_fullsized":"2024-01-15T11:25:16.272390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.263680+0000","last_change":"2024-01-15T11:25:16.259468+0000","last_active":"2024-01-15T11:25:16.263680+0000","last_peered":"2024-01-15T11:25:16.263680+0000","last_clean":"2024-01-15T11:25:16.263680+0000","last_became_active":"2024-01-15T11:25:16.258703+0000","last_became_peered":"2024-01-15T11:25:16.258703+0000","last_unstale":"2024-01-15T11:25:16.263680+0000","last_undegraded":"2024-01-15T11:25:16.263680+0000","last_fullsized":"2024-01-15T11:25:16.263680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.273292+0000","last_change":"2024-01-15T11:25:16.266445+0000","last_active":"2024-01-15T11:25:16.273292+0000","last_peered":"2024-01-15T11:25:16.273292+0000","last_clean":"2024-01-15T11:25:16.273292+0000","last_became_active":"2024-01-15T11:25:16.265486+0000","last_became_peered":"2024-01-15T11:25:16.265486+0000","last_unstale":"2024-01-15T11:25:16.273292+0000","last_undegraded":"2024-01-15T11:25:16.273292+0000","last_fullsized":"2024-01-15T11:25:16.273292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:17:27.965755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.261004+0000","last_change":"2024-01-15T11:25:16.259210+0000","last_active":"2024-01-15T11:25:16.261004+0000","last_peered":"2024-01-15T11:25:16.261004+0000","last_clean":"2024-01-15T11:25:16.261004+0000","last_became_active":"2024-01-15T11:25:16.258919+0000","last_became_peered":"2024-01-15T11:25:16.258919+0000","last_unstale":"2024-01-15T11:25:16.261004+0000","last_undegraded":"2024-01-15T11:25:16.261004+0000","last_fullsized":"2024-01-15T11:25:16.261004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:59.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.266744+0000","last_change":"2024-01-15T11:25:16.264604+0000","last_active":"2024-01-15T11:25:16.266744+0000","last_peered":"2024-01-15T11:25:16.266744+0000","last_clean":"2024-01-15T11:25:16.266744+0000","last_became_active":"2024-01-15T11:25:16.263950+0000","last_became_peered":"2024-01-15T11:25:16.263950+0000","last_unstale":"2024-01-15T11:25:16.266744+0000","last_undegraded":"2024-01-15T11:25:16.266744+0000","last_fullsized":"2024-01-15T11:25:16.266744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:43.332244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260084+0000","last_change":"2024-01-15T11:25:16.256747+0000","last_active":"2024-01-15T11:25:16.260084+0000","last_peered":"2024-01-15T11:25:16.260084+0000","last_clean":"2024-01-15T11:25:16.260084+0000","last_became_active":"2024-01-15T11:25:16.256464+0000","last_became_peered":"2024-01-15T11:25:16.256464+0000","last_unstale":"2024-01-15T11:25:16.260084+0000","last_undegraded":"2024-01-15T11:25:16.260084+0000","last_fullsized":"2024-01-15T11:25:16.260084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272891+0000","last_change":"2024-01-15T11:25:16.268099+0000","last_active":"2024-01-15T11:25:16.272891+0000","last_peered":"2024-01-15T11:25:16.272891+0000","last_clean":"2024-01-15T11:25:16.272891+0000","last_became_active":"2024-01-15T11:25:16.267826+0000","last_became_peered":"2024-01-15T11:25:16.267826+0000","last_unstale":"2024-01-15T11:25:16.272891+0000","last_undegraded":"2024-01-15T11:25:16.272891+0000","last_fullsized":"2024-01-15T11:25:16.272891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.265130+0000","last_change":"2024-01-15T11:25:16.263641+0000","last_active":"2024-01-15T11:25:16.265130+0000","last_peered":"2024-01-15T11:25:16.265130+0000","last_clean":"2024-01-15T11:25:16.265130+0000","last_became_active":"2024-01-15T11:25:16.263368+0000","last_became_peered":"2024-01-15T11:25:16.263368+0000","last_unstale":"2024-01-15T11:25:16.265130+0000","last_undegraded":"2024-01-15T11:25:16.265130+0000","last_fullsized":"2024-01-15T11:25:16.265130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:05.228308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.268937+0000","last_change":"2024-01-15T11:25:16.262170+0000","last_active":"2024-01-15T11:25:16.268937+0000","last_peered":"2024-01-15T11:25:16.268937+0000","last_clean":"2024-01-15T11:25:16.268937+0000","last_became_active":"2024-01-15T11:25:16.261695+0000","last_became_peered":"2024-01-15T11:25:16.261695+0000","last_unstale":"2024-01-15T11:25:16.268937+0000","last_undegraded":"2024-01-15T11:25:16.268937+0000","last_fullsized":"2024-01-15T11:25:16.268937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.267627+0000","last_change":"2024-01-15T11:25:16.265406+0000","last_active":"2024-01-15T11:25:16.267627+0000","last_peered":"2024-01-15T11:25:16.267627+0000","last_clean":"2024-01-15T11:25:16.267627+0000","last_became_active":"2024-01-15T11:25:16.265150+0000","last_became_peered":"2024-01-15T11:25:16.265150+0000","last_unstale":"2024-01-15T11:25:16.267627+0000","last_undegraded":"2024-01-15T11:25:16.267627+0000","last_fullsized":"2024-01-15T11:25:16.267627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:19:46.378784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275073+0000","last_change":"2024-01-15T11:25:16.271625+0000","last_active":"2024-01-15T11:25:16.275073+0000","last_peered":"2024-01-15T11:25:16.275073+0000","last_clean":"2024-01-15T11:25:16.275073+0000","last_became_active":"2024-01-15T11:25:16.271361+0000","last_became_peered":"2024-01-15T11:25:16.271361+0000","last_unstale":"2024-01-15T11:25:16.275073+0000","last_undegraded":"2024-01-15T11:25:16.275073+0000","last_fullsized":"2024-01-15T11:25:16.275073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:02:52.680406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272252+0000","last_change":"2024-01-15T11:25:16.271113+0000","last_active":"2024-01-15T11:25:16.272252+0000","last_peered":"2024-01-15T11:25:16.272252+0000","last_clean":"2024-01-15T11:25:16.272252+0000","last_became_active":"2024-01-15T11:25:16.270795+0000","last_became_peered":"2024-01-15T11:25:16.270795+0000","last_unstale":"2024-01-15T11:25:16.272252+0000","last_undegraded":"2024-01-15T11:25:16.272252+0000","last_fullsized":"2024-01-15T11:25:16.272252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.431909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272566+0000","last_change":"2024-01-15T11:25:16.262518+0000","last_active":"2024-01-15T11:25:16.272566+0000","last_peered":"2024-01-15T11:25:16.272566+0000","last_clean":"2024-01-15T11:25:16.272566+0000","last_became_active":"2024-01-15T11:25:16.262049+0000","last_became_peered":"2024-01-15T11:25:16.262049+0000","last_unstale":"2024-01-15T11:25:16.272566+0000","last_undegraded":"2024-01-15T11:25:16.272566+0000","last_fullsized":"2024-01-15T11:25:16.272566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.431909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":66222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":64770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":774144,"data_stored":648203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":66222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":64736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":64700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T11:25:20.350 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T11:25:20.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.509+0000 7f59be269700 1 Processor -- start 2024-01-15T11:25:20.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.509+0000 7f59be269700 1 -- start start 2024-01-15T11:25:20.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.510+0000 7f59be269700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59b815af10 0x7f59b815e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.510+0000 7f59be269700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b80ce140 0x7f59b815ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.510+0000 7f59be269700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b81414c0 con 0x7f59b815f460 2024-01-15T11:25:20.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.510+0000 7f59be269700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b815f8c0 con 0x7f59b815af10 2024-01-15T11:25:20.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.510+0000 7f59be269700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b80ce9f0 con 0x7f59b80ce140 2024-01-15T11:25:20.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b7fff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59b815af10 0x7f59b815e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b77fe700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b80ce140 0x7f59b815ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b7fff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59b815af10 0x7f59b815e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51222/0 (socket says 172.21.15.113:51222) 2024-01-15T11:25:20.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b7fff700 1 -- 172.21.15.113:0/2170425365 learned_addr learned my addr 172.21.15.113:0/2170425365 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:20.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59bc806700 1 --1- 172.21.15.113:0/2170425365 >> v1:172.21.15.141:6789/0 conn(0x7f59b815f460 0x7f59b815b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:35220/0 (socket says 172.21.15.113:35220) 2024-01-15T11:25:20.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b7fff700 1 -- 172.21.15.113:0/2170425365 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b80ce140 msgr2=0x7f59b815ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:20.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b7fff700 1 --2- 172.21.15.113:0/2170425365 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b80ce140 0x7f59b815ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b7fff700 1 -- 172.21.15.113:0/2170425365 >> v1:172.21.15.141:6789/0 conn(0x7f59b815f460 legacy=0x7f59b815b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b7fff700 1 -- 172.21.15.113:0/2170425365 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b80cecd0 con 0x7f59b815af10 2024-01-15T11:25:20.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b77fe700 1 --2- 172.21.15.113:0/2170425365 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b80ce140 0x7f59b815ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:20.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b7fff700 1 --2- 172.21.15.113:0/2170425365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b815af10 0x7f59b815e9e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f59a80098f0 tx=0x7f59a8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe2b0cdf800bedf4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:20.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b6ffd700 1 -- 172.21.15.113:0/2170425365 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a8011830 con 0x7f59b815af10 2024-01-15T11:25:20.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b6ffd700 1 -- 172.21.15.113:0/2170425365 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a8011990 con 0x7f59b815af10 2024-01-15T11:25:20.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.511+0000 7f59b6ffd700 1 -- 172.21.15.113:0/2170425365 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a8011d20 con 0x7f59b815af10 2024-01-15T11:25:20.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.512+0000 7f59be269700 1 -- 172.21.15.113:0/2170425365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b815af10 msgr2=0x7f59b815e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.512+0000 7f59be269700 1 --2- 172.21.15.113:0/2170425365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b815af10 0x7f59b815e9e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f59a80098f0 tx=0x7f59a8009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:20.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.512+0000 7f59be269700 1 -- 172.21.15.113:0/2170425365 shutdown_connections 2024-01-15T11:25:20.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.512+0000 7f59be269700 1 --2- 172.21.15.113:0/2170425365 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b815af10 0x7f59b815e9e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.512+0000 7f59be269700 1 --2- 172.21.15.113:0/2170425365 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b80ce140 0x7f59b815ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.512+0000 7f59be269700 1 -- 172.21.15.113:0/2170425365 >> 172.21.15.113:0/2170425365 conn(0x7f59b80cd570 msgr2=0x7f59b80cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:20.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.512+0000 7f59be269700 1 -- 172.21.15.113:0/2170425365 shutdown_connections 2024-01-15T11:25:20.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.512+0000 7f59be269700 1 -- 172.21.15.113:0/2170425365 wait complete. 2024-01-15T11:25:20.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.513+0000 7f59be269700 1 Processor -- start 2024-01-15T11:25:20.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.513+0000 7f59be269700 1 -- start start 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59be269700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b80ce140 0x7f59b81f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59be269700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59b815af10 0x7f59b81445a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59be269700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b815f460 0x7f59b8159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59be269700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b8144ae0 con 0x7f59b815af10 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59be269700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b81f5620 con 0x7f59b80ce140 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59be269700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b81f5be0 con 0x7f59b815f460 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59b77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59b815af10 0x7f59b81445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59b77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59b815af10 0x7f59b81445a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45498/0 (socket says 172.21.15.113:45498) 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59b77fe700 1 -- 172.21.15.113:0/1380022575 learned_addr learned my addr 172.21.15.113:0/1380022575 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59bc806700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b815f460 0x7f59b8159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59b7fff700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b80ce140 0x7f59b81f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59b77fe700 1 -- 172.21.15.113:0/1380022575 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b815f460 msgr2=0x7f59b8159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59b77fe700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b815f460 0x7f59b8159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.514+0000 7f59b77fe700 1 -- 172.21.15.113:0/1380022575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b80ce140 msgr2=0x7f59b81f6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.515+0000 7f59b77fe700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b80ce140 0x7f59b81f6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.515+0000 7f59b77fe700 1 -- 172.21.15.113:0/1380022575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ac0094a0 con 0x7f59b815af10 2024-01-15T11:25:20.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.515+0000 7f59b77fe700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59b815af10 0x7f59b81445a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59ac00ec10 tx=0x7f59ac00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:20.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.515+0000 7f59b4ff9700 1 -- 172.21.15.113:0/1380022575 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0097a0 con 0x7f59b815af10 2024-01-15T11:25:20.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.515+0000 7f59b4ff9700 1 -- 172.21.15.113:0/1380022575 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ac00c850 con 0x7f59b815af10 2024-01-15T11:25:20.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.515+0000 7f59b4ff9700 1 -- 172.21.15.113:0/1380022575 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ac00cbe0 con 0x7f59b815af10 2024-01-15T11:25:20.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.515+0000 7f59be269700 1 -- 172.21.15.113:0/1380022575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a8009550 con 0x7f59b815af10 2024-01-15T11:25:20.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.515+0000 7f59be269700 1 -- 172.21.15.113:0/1380022575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b8143db0 con 0x7f59b815af10 2024-01-15T11:25:20.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.516+0000 7f59be269700 1 -- 172.21.15.113:0/1380022575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b8067f60 con 0x7f59b815af10 2024-01-15T11:25:20.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.516+0000 7f59b4ff9700 1 -- 172.21.15.113:0/1380022575 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f59ac00cd40 con 0x7f59b815af10 2024-01-15T11:25:20.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.516+0000 7f59b4ff9700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f59a406b350 0x7f59a406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.517+0000 7f59b4ff9700 1 -- 172.21.15.113:0/1380022575 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59ac0897a0 con 0x7f59b815af10 2024-01-15T11:25:20.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.517+0000 7f59b7fff700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f59a406b350 0x7f59a406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.517+0000 7f59b7fff700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f59a406b350 0x7f59a406d810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f59a8009c40 tx=0x7f59a8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:20.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.521+0000 7f59b4ff9700 1 -- 172.21.15.113:0/1380022575 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59ac0546a0 con 0x7f59b815af10 2024-01-15T11:25:20.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.707+0000 7f59be269700 1 -- 172.21.15.113:0/1380022575 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59b80686e0 con 0x7f59a406b350 2024-01-15T11:25:20.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.714+0000 7f59b4ff9700 1 -- 172.21.15.113:0/1380022575 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264603 (secure 0 0 0) 0x7f59b80686e0 con 0x7f59a406b350 2024-01-15T11:25:20.716 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.718+0000 7f59be269700 1 -- 172.21.15.113:0/1380022575 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f59a406b350 msgr2=0x7f59a406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.718+0000 7f59be269700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f59a406b350 0x7f59a406d810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f59a8009c40 tx=0x7f59a8008030 comp rx=0 tx=0).stop 2024-01-15T11:25:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.718+0000 7f59be269700 1 -- 172.21.15.113:0/1380022575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59b815af10 msgr2=0x7f59b81445a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.718+0000 7f59be269700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59b815af10 0x7f59b81445a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59ac00ec10 tx=0x7f59ac00ef20 comp rx=0 tx=0).stop 2024-01-15T11:25:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.718+0000 7f59be269700 1 -- 172.21.15.113:0/1380022575 shutdown_connections 2024-01-15T11:25:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.718+0000 7f59be269700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f59a406b350 0x7f59a406d810 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.718+0000 7f59be269700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b815f460 0x7f59b8159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.718+0000 7f59be269700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b80ce140 0x7f59b81f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.718+0000 7f59be269700 1 --2- 172.21.15.113:0/1380022575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59b815af10 0x7f59b81445a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.718+0000 7f59be269700 1 -- 172.21.15.113:0/1380022575 >> 172.21.15.113:0/1380022575 conn(0x7f59b80cd570 msgr2=0x7f59b806dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.718+0000 7f59be269700 1 -- 172.21.15.113:0/1380022575 shutdown_connections 2024-01-15T11:25:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.718+0000 7f59be269700 1 -- 172.21.15.113:0/1380022575 wait complete. 2024-01-15T11:25:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-15T11:25:20.728 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-15T11:25:20.109581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218888,"kb_used_data":2704,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755832,"statfs":{"total":773094113280,"available":772869971968,"internally_reserved":0,"allocated":2768896,"data_stored":1695023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.894917"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.273391+0000","last_change":"2024-01-15T11:25:16.269790+0000","last_active":"2024-01-15T11:25:16.273391+0000","last_peered":"2024-01-15T11:25:16.273391+0000","last_clean":"2024-01-15T11:25:16.273391+0000","last_became_active":"2024-01-15T11:25:16.269540+0000","last_became_peered":"2024-01-15T11:25:16.269540+0000","last_unstale":"2024-01-15T11:25:16.273391+0000","last_undegraded":"2024-01-15T11:25:16.273391+0000","last_fullsized":"2024-01-15T11:25:16.273391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:02:52.680406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260233+0000","last_change":"2024-01-15T11:25:16.254768+0000","last_active":"2024-01-15T11:25:16.260233+0000","last_peered":"2024-01-15T11:25:16.260233+0000","last_clean":"2024-01-15T11:25:16.260233+0000","last_became_active":"2024-01-15T11:25:16.254438+0000","last_became_peered":"2024-01-15T11:25:16.254438+0000","last_unstale":"2024-01-15T11:25:16.260233+0000","last_undegraded":"2024-01-15T11:25:16.260233+0000","last_fullsized":"2024-01-15T11:25:16.260233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260272+0000","last_change":"2024-01-15T11:25:16.255993+0000","last_active":"2024-01-15T11:25:16.260272+0000","last_peered":"2024-01-15T11:25:16.260272+0000","last_clean":"2024-01-15T11:25:16.260272+0000","last_became_active":"2024-01-15T11:25:16.255718+0000","last_became_peered":"2024-01-15T11:25:16.255718+0000","last_unstale":"2024-01-15T11:25:16.260272+0000","last_undegraded":"2024-01-15T11:25:16.260272+0000","last_fullsized":"2024-01-15T11:25:16.260272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.268918+0000","last_change":"2024-01-15T11:25:16.262620+0000","last_active":"2024-01-15T11:25:16.268918+0000","last_peered":"2024-01-15T11:25:16.268918+0000","last_clean":"2024-01-15T11:25:16.268918+0000","last_became_active":"2024-01-15T11:25:16.262147+0000","last_became_peered":"2024-01-15T11:25:16.262147+0000","last_unstale":"2024-01-15T11:25:16.268918+0000","last_undegraded":"2024-01-15T11:25:16.268918+0000","last_fullsized":"2024-01-15T11:25:16.268918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:26.801860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.266228+0000","last_change":"2024-01-15T11:25:16.252826+0000","last_active":"2024-01-15T11:25:16.266228+0000","last_peered":"2024-01-15T11:25:16.266228+0000","last_clean":"2024-01-15T11:25:16.266228+0000","last_became_active":"2024-01-15T11:25:16.251976+0000","last_became_peered":"2024-01-15T11:25:16.251976+0000","last_unstale":"2024-01-15T11:25:16.266228+0000","last_undegraded":"2024-01-15T11:25:16.266228+0000","last_fullsized":"2024-01-15T11:25:16.266228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:17:27.965755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.273315+0000","last_change":"2024-01-15T11:25:16.266650+0000","last_active":"2024-01-15T11:25:16.273315+0000","last_peered":"2024-01-15T11:25:16.273315+0000","last_clean":"2024-01-15T11:25:16.273315+0000","last_became_active":"2024-01-15T11:25:16.266157+0000","last_became_peered":"2024-01-15T11:25:16.266157+0000","last_unstale":"2024-01-15T11:25:16.273315+0000","last_undegraded":"2024-01-15T11:25:16.273315+0000","last_fullsized":"2024-01-15T11:25:16.273315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:56:33.010927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.266725+0000","last_change":"2024-01-15T11:25:16.255535+0000","last_active":"2024-01-15T11:25:16.266725+0000","last_peered":"2024-01-15T11:25:16.266725+0000","last_clean":"2024-01-15T11:25:16.266725+0000","last_became_active":"2024-01-15T11:25:16.255248+0000","last_became_peered":"2024-01-15T11:25:16.255248+0000","last_unstale":"2024-01-15T11:25:16.266725+0000","last_undegraded":"2024-01-15T11:25:16.266725+0000","last_fullsized":"2024-01-15T11:25:16.266725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:26.586428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260526+0000","last_change":"2024-01-15T11:25:16.258135+0000","last_active":"2024-01-15T11:25:16.260526+0000","last_peered":"2024-01-15T11:25:16.260526+0000","last_clean":"2024-01-15T11:25:16.260526+0000","last_became_active":"2024-01-15T11:25:16.257859+0000","last_became_peered":"2024-01-15T11:25:16.257859+0000","last_unstale":"2024-01-15T11:25:16.260526+0000","last_undegraded":"2024-01-15T11:25:16.260526+0000","last_fullsized":"2024-01-15T11:25:16.260526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:26.386576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.270443+0000","last_change":"2024-01-15T11:25:16.261234+0000","last_active":"2024-01-15T11:25:16.270443+0000","last_peered":"2024-01-15T11:25:16.270443+0000","last_clean":"2024-01-15T11:25:16.270443+0000","last_became_active":"2024-01-15T11:25:16.260952+0000","last_became_peered":"2024-01-15T11:25:16.260952+0000","last_unstale":"2024-01-15T11:25:16.270443+0000","last_undegraded":"2024-01-15T11:25:16.270443+0000","last_fullsized":"2024-01-15T11:25:16.270443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:15:35.970176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272545+0000","last_change":"2024-01-15T11:25:16.266369+0000","last_active":"2024-01-15T11:25:16.272545+0000","last_peered":"2024-01-15T11:25:16.272545+0000","last_clean":"2024-01-15T11:25:16.272545+0000","last_became_active":"2024-01-15T11:25:16.266091+0000","last_became_peered":"2024-01-15T11:25:16.266091+0000","last_unstale":"2024-01-15T11:25:16.272545+0000","last_undegraded":"2024-01-15T11:25:16.272545+0000","last_fullsized":"2024-01-15T11:25:16.272545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:05.228308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272409+0000","last_change":"2024-01-15T11:25:16.259582+0000","last_active":"2024-01-15T11:25:16.272409+0000","last_peered":"2024-01-15T11:25:16.272409+0000","last_clean":"2024-01-15T11:25:16.272409+0000","last_became_active":"2024-01-15T11:25:16.259270+0000","last_became_peered":"2024-01-15T11:25:16.259270+0000","last_unstale":"2024-01-15T11:25:16.272409+0000","last_undegraded":"2024-01-15T11:25:16.272409+0000","last_fullsized":"2024-01-15T11:25:16.272409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275816+0000","last_change":"2024-01-15T11:25:16.274052+0000","last_active":"2024-01-15T11:25:16.275816+0000","last_peered":"2024-01-15T11:25:16.275816+0000","last_clean":"2024-01-15T11:25:16.275816+0000","last_became_active":"2024-01-15T11:25:16.273798+0000","last_became_peered":"2024-01-15T11:25:16.273798+0000","last_unstale":"2024-01-15T11:25:16.275816+0000","last_undegraded":"2024-01-15T11:25:16.275816+0000","last_fullsized":"2024-01-15T11:25:16.275816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:19:46.378784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.253156+0000","last_change":"2024-01-15T11:25:16.250458+0000","last_active":"2024-01-15T11:25:16.253156+0000","last_peered":"2024-01-15T11:25:16.253156+0000","last_clean":"2024-01-15T11:25:16.253156+0000","last_became_active":"2024-01-15T11:25:16.250262+0000","last_became_peered":"2024-01-15T11:25:16.250262+0000","last_unstale":"2024-01-15T11:25:16.253156+0000","last_undegraded":"2024-01-15T11:25:16.253156+0000","last_fullsized":"2024-01-15T11:25:16.253156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:33.426251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.258679+0000","last_change":"2024-01-15T11:25:16.253950+0000","last_active":"2024-01-15T11:25:16.258679+0000","last_peered":"2024-01-15T11:25:16.258679+0000","last_clean":"2024-01-15T11:25:16.258679+0000","last_became_active":"2024-01-15T11:25:16.253667+0000","last_became_peered":"2024-01-15T11:25:16.253667+0000","last_unstale":"2024-01-15T11:25:16.258679+0000","last_undegraded":"2024-01-15T11:25:16.258679+0000","last_fullsized":"2024-01-15T11:25:16.258679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:13.140323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.269756+0000","last_change":"2024-01-15T11:25:16.268387+0000","last_active":"2024-01-15T11:25:16.269756+0000","last_peered":"2024-01-15T11:25:16.269756+0000","last_clean":"2024-01-15T11:25:16.269756+0000","last_became_active":"2024-01-15T11:25:16.268088+0000","last_became_peered":"2024-01-15T11:25:16.268088+0000","last_unstale":"2024-01-15T11:25:16.269756+0000","last_undegraded":"2024-01-15T11:25:16.269756+0000","last_fullsized":"2024-01-15T11:25:16.269756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:59.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.266691+0000","last_change":"2024-01-15T11:25:16.261989+0000","last_active":"2024-01-15T11:25:16.266691+0000","last_peered":"2024-01-15T11:25:16.266691+0000","last_clean":"2024-01-15T11:25:16.266691+0000","last_became_active":"2024-01-15T11:25:16.261260+0000","last_became_peered":"2024-01-15T11:25:16.261260+0000","last_unstale":"2024-01-15T11:25:16.266691+0000","last_undegraded":"2024-01-15T11:25:16.266691+0000","last_fullsized":"2024-01-15T11:25:16.266691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.274834+0000","last_change":"2024-01-15T11:25:16.270904+0000","last_active":"2024-01-15T11:25:16.274834+0000","last_peered":"2024-01-15T11:25:16.274834+0000","last_clean":"2024-01-15T11:25:16.274834+0000","last_became_active":"2024-01-15T11:25:16.270611+0000","last_became_peered":"2024-01-15T11:25:16.270611+0000","last_unstale":"2024-01-15T11:25:16.274834+0000","last_undegraded":"2024-01-15T11:25:16.274834+0000","last_fullsized":"2024-01-15T11:25:16.274834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:54.759984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.261115+0000","last_change":"2024-01-15T11:25:16.257019+0000","last_active":"2024-01-15T11:25:16.261115+0000","last_peered":"2024-01-15T11:25:16.261115+0000","last_clean":"2024-01-15T11:25:16.261115+0000","last_became_active":"2024-01-15T11:25:16.256278+0000","last_became_peered":"2024-01-15T11:25:16.256278+0000","last_unstale":"2024-01-15T11:25:16.261115+0000","last_undegraded":"2024-01-15T11:25:16.261115+0000","last_fullsized":"2024-01-15T11:25:16.261115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.251622+0000","last_change":"2024-01-15T11:25:16.248798+0000","last_active":"2024-01-15T11:25:16.251622+0000","last_peered":"2024-01-15T11:25:16.251622+0000","last_clean":"2024-01-15T11:25:16.251622+0000","last_became_active":"2024-01-15T11:25:16.248488+0000","last_became_peered":"2024-01-15T11:25:16.248488+0000","last_unstale":"2024-01-15T11:25:16.251622+0000","last_undegraded":"2024-01-15T11:25:16.251622+0000","last_fullsized":"2024-01-15T11:25:16.251622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:26.386576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.276047+0000","last_change":"2024-01-15T11:25:15.233384+0000","last_active":"2024-01-15T11:25:16.276047+0000","last_peered":"2024-01-15T11:25:16.276047+0000","last_clean":"2024-01-15T11:25:16.276047+0000","last_became_active":"2024-01-15T11:25:15.233134+0000","last_became_peered":"2024-01-15T11:25:15.233134+0000","last_unstale":"2024-01-15T11:25:16.276047+0000","last_undegraded":"2024-01-15T11:25:16.276047+0000","last_fullsized":"2024-01-15T11:25:16.276047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260826+0000","last_change":"2024-01-15T11:25:16.256579+0000","last_active":"2024-01-15T11:25:16.260826+0000","last_peered":"2024-01-15T11:25:16.260826+0000","last_clean":"2024-01-15T11:25:16.260826+0000","last_became_active":"2024-01-15T11:25:16.256320+0000","last_became_peered":"2024-01-15T11:25:16.256320+0000","last_unstale":"2024-01-15T11:25:16.260826+0000","last_undegraded":"2024-01-15T11:25:16.260826+0000","last_fullsized":"2024-01-15T11:25:16.260826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.271304+0000","last_change":"2024-01-15T11:25:15.230417+0000","last_active":"2024-01-15T11:25:16.271304+0000","last_peered":"2024-01-15T11:25:16.271304+0000","last_clean":"2024-01-15T11:25:16.271304+0000","last_became_active":"2024-01-15T11:25:15.230135+0000","last_became_peered":"2024-01-15T11:25:15.230135+0000","last_unstale":"2024-01-15T11:25:16.271304+0000","last_undegraded":"2024-01-15T11:25:16.271304+0000","last_fullsized":"2024-01-15T11:25:16.271304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:25.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.263184+0000","last_change":"2024-01-15T11:25:16.259985+0000","last_active":"2024-01-15T11:25:16.263184+0000","last_peered":"2024-01-15T11:25:16.263184+0000","last_clean":"2024-01-15T11:25:16.263184+0000","last_became_active":"2024-01-15T11:25:16.259697+0000","last_became_peered":"2024-01-15T11:25:16.259697+0000","last_unstale":"2024-01-15T11:25:16.263184+0000","last_undegraded":"2024-01-15T11:25:16.263184+0000","last_fullsized":"2024-01-15T11:25:16.263184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:26.586428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272821+0000","last_change":"2024-01-15T11:25:15.238695+0000","last_active":"2024-01-15T11:25:16.272821+0000","last_peered":"2024-01-15T11:25:16.272821+0000","last_clean":"2024-01-15T11:25:16.272821+0000","last_became_active":"2024-01-15T11:25:15.238377+0000","last_became_peered":"2024-01-15T11:25:15.238377+0000","last_unstale":"2024-01-15T11:25:16.272821+0000","last_undegraded":"2024-01-15T11:25:16.272821+0000","last_fullsized":"2024-01-15T11:25:16.272821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:48.780156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.268942+0000","last_change":"2024-01-15T11:25:16.264379+0000","last_active":"2024-01-15T11:25:16.268942+0000","last_peered":"2024-01-15T11:25:16.268942+0000","last_clean":"2024-01-15T11:25:16.268942+0000","last_became_active":"2024-01-15T11:25:16.264121+0000","last_became_peered":"2024-01-15T11:25:16.264121+0000","last_unstale":"2024-01-15T11:25:16.268942+0000","last_undegraded":"2024-01-15T11:25:16.268942+0000","last_fullsized":"2024-01-15T11:25:16.268942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:17.504194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275466+0000","last_change":"2024-01-15T11:25:15.237423+0000","last_active":"2024-01-15T11:25:16.275466+0000","last_peered":"2024-01-15T11:25:16.275466+0000","last_clean":"2024-01-15T11:25:16.275466+0000","last_became_active":"2024-01-15T11:25:15.237121+0000","last_became_peered":"2024-01-15T11:25:15.237121+0000","last_unstale":"2024-01-15T11:25:16.275466+0000","last_undegraded":"2024-01-15T11:25:16.275466+0000","last_fullsized":"2024-01-15T11:25:16.275466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:04.223510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.273258+0000","last_change":"2024-01-15T11:25:16.266842+0000","last_active":"2024-01-15T11:25:16.273258+0000","last_peered":"2024-01-15T11:25:16.273258+0000","last_clean":"2024-01-15T11:25:16.273258+0000","last_became_active":"2024-01-15T11:25:16.266606+0000","last_became_peered":"2024-01-15T11:25:16.266606+0000","last_unstale":"2024-01-15T11:25:16.273258+0000","last_undegraded":"2024-01-15T11:25:16.273258+0000","last_fullsized":"2024-01-15T11:25:16.273258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275555+0000","last_change":"2024-01-15T11:25:15.232263+0000","last_active":"2024-01-15T11:25:16.275555+0000","last_peered":"2024-01-15T11:25:16.275555+0000","last_clean":"2024-01-15T11:25:16.275555+0000","last_became_active":"2024-01-15T11:25:15.231976+0000","last_became_peered":"2024-01-15T11:25:15.231976+0000","last_unstale":"2024-01-15T11:25:16.275555+0000","last_undegraded":"2024-01-15T11:25:16.275555+0000","last_fullsized":"2024-01-15T11:25:16.275555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:25.797063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.268081+0000","last_change":"2024-01-15T11:25:16.263430+0000","last_active":"2024-01-15T11:25:16.268081+0000","last_peered":"2024-01-15T11:25:16.268081+0000","last_clean":"2024-01-15T11:25:16.268081+0000","last_became_active":"2024-01-15T11:25:16.261944+0000","last_became_peered":"2024-01-15T11:25:16.261944+0000","last_unstale":"2024-01-15T11:25:16.268081+0000","last_undegraded":"2024-01-15T11:25:16.268081+0000","last_fullsized":"2024-01-15T11:25:16.268081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:13.140323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275875+0000","last_change":"2024-01-15T11:25:15.235451+0000","last_active":"2024-01-15T11:25:16.275875+0000","last_peered":"2024-01-15T11:25:16.275875+0000","last_clean":"2024-01-15T11:25:16.275875+0000","last_became_active":"2024-01-15T11:25:15.235166+0000","last_became_peered":"2024-01-15T11:25:15.235166+0000","last_unstale":"2024-01-15T11:25:16.275875+0000","last_undegraded":"2024-01-15T11:25:16.275875+0000","last_fullsized":"2024-01-15T11:25:16.275875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.263710+0000","last_change":"2024-01-15T11:25:16.262932+0000","last_active":"2024-01-15T11:25:16.263710+0000","last_peered":"2024-01-15T11:25:16.263710+0000","last_clean":"2024-01-15T11:25:16.263710+0000","last_became_active":"2024-01-15T11:25:16.262667+0000","last_became_peered":"2024-01-15T11:25:16.262667+0000","last_unstale":"2024-01-15T11:25:16.263710+0000","last_undegraded":"2024-01-15T11:25:16.263710+0000","last_fullsized":"2024-01-15T11:25:16.263710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:19:46.378784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.244597+0000","last_change":"2024-01-15T11:25:15.232286+0000","last_active":"2024-01-15T11:25:16.244597+0000","last_peered":"2024-01-15T11:25:16.244597+0000","last_clean":"2024-01-15T11:25:16.244597+0000","last_became_active":"2024-01-15T11:25:15.230764+0000","last_became_peered":"2024-01-15T11:25:15.230764+0000","last_unstale":"2024-01-15T11:25:16.244597+0000","last_undegraded":"2024-01-15T11:25:16.244597+0000","last_fullsized":"2024-01-15T11:25:16.244597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:53.755186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260488+0000","last_change":"2024-01-15T11:25:16.254427+0000","last_active":"2024-01-15T11:25:16.260488+0000","last_peered":"2024-01-15T11:25:16.260488+0000","last_clean":"2024-01-15T11:25:16.260488+0000","last_became_active":"2024-01-15T11:25:16.253271+0000","last_became_peered":"2024-01-15T11:25:16.253271+0000","last_unstale":"2024-01-15T11:25:16.260488+0000","last_undegraded":"2024-01-15T11:25:16.260488+0000","last_fullsized":"2024-01-15T11:25:16.260488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.431909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.250014+0000","last_change":"2024-01-15T11:25:15.231253+0000","last_active":"2024-01-15T11:25:16.250014+0000","last_peered":"2024-01-15T11:25:16.250014+0000","last_clean":"2024-01-15T11:25:16.250014+0000","last_became_active":"2024-01-15T11:25:15.230935+0000","last_became_peered":"2024-01-15T11:25:15.230935+0000","last_unstale":"2024-01-15T11:25:16.250014+0000","last_undegraded":"2024-01-15T11:25:16.250014+0000","last_fullsized":"2024-01-15T11:25:16.250014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:04.223510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260112+0000","last_change":"2024-01-15T11:25:16.256798+0000","last_active":"2024-01-15T11:25:16.260112+0000","last_peered":"2024-01-15T11:25:16.260112+0000","last_clean":"2024-01-15T11:25:16.260112+0000","last_became_active":"2024-01-15T11:25:16.256540+0000","last_became_peered":"2024-01-15T11:25:16.256540+0000","last_unstale":"2024-01-15T11:25:16.260112+0000","last_undegraded":"2024-01-15T11:25:16.260112+0000","last_fullsized":"2024-01-15T11:25:16.260112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.431909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.251444+0000","last_change":"2024-01-15T11:25:15.236440+0000","last_active":"2024-01-15T11:25:16.251444+0000","last_peered":"2024-01-15T11:25:16.251444+0000","last_clean":"2024-01-15T11:25:16.251444+0000","last_became_active":"2024-01-15T11:25:15.236034+0000","last_became_peered":"2024-01-15T11:25:15.236034+0000","last_unstale":"2024-01-15T11:25:16.251444+0000","last_undegraded":"2024-01-15T11:25:16.251444+0000","last_fullsized":"2024-01-15T11:25:16.251444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:25.797063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.274623+0000","last_change":"2024-01-15T11:25:16.270509+0000","last_active":"2024-01-15T11:25:16.274623+0000","last_peered":"2024-01-15T11:25:16.274623+0000","last_clean":"2024-01-15T11:25:16.274623+0000","last_became_active":"2024-01-15T11:25:16.270319+0000","last_became_peered":"2024-01-15T11:25:16.270319+0000","last_unstale":"2024-01-15T11:25:16.274623+0000","last_undegraded":"2024-01-15T11:25:16.274623+0000","last_fullsized":"2024-01-15T11:25:16.274623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:59.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.252880+0000","last_change":"2024-01-15T11:25:15.235314+0000","last_active":"2024-01-15T11:25:16.252880+0000","last_peered":"2024-01-15T11:25:16.252880+0000","last_clean":"2024-01-15T11:25:16.252880+0000","last_became_active":"2024-01-15T11:25:15.232515+0000","last_became_peered":"2024-01-15T11:25:15.232515+0000","last_unstale":"2024-01-15T11:25:16.252880+0000","last_undegraded":"2024-01-15T11:25:16.252880+0000","last_fullsized":"2024-01-15T11:25:16.252880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:32.421454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.263204+0000","last_change":"2024-01-15T11:25:16.260223+0000","last_active":"2024-01-15T11:25:16.263204+0000","last_peered":"2024-01-15T11:25:16.263204+0000","last_clean":"2024-01-15T11:25:16.263204+0000","last_became_active":"2024-01-15T11:25:16.259365+0000","last_became_peered":"2024-01-15T11:25:16.259365+0000","last_unstale":"2024-01-15T11:25:16.263204+0000","last_undegraded":"2024-01-15T11:25:16.263204+0000","last_fullsized":"2024-01-15T11:25:16.263204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.273490+0000","last_change":"2024-01-15T11:25:15.233433+0000","last_active":"2024-01-15T11:25:16.273490+0000","last_peered":"2024-01-15T11:25:16.273490+0000","last_clean":"2024-01-15T11:25:16.273490+0000","last_became_active":"2024-01-15T11:25:15.233046+0000","last_became_peered":"2024-01-15T11:25:15.233046+0000","last_unstale":"2024-01-15T11:25:16.273490+0000","last_undegraded":"2024-01-15T11:25:16.273490+0000","last_fullsized":"2024-01-15T11:25:16.273490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:25.797063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.277312+0000","last_change":"2024-01-15T11:25:16.275023+0000","last_active":"2024-01-15T11:25:16.277312+0000","last_peered":"2024-01-15T11:25:16.277312+0000","last_clean":"2024-01-15T11:25:16.277312+0000","last_became_active":"2024-01-15T11:25:16.274744+0000","last_became_peered":"2024-01-15T11:25:16.274744+0000","last_unstale":"2024-01-15T11:25:16.277312+0000","last_undegraded":"2024-01-15T11:25:16.277312+0000","last_fullsized":"2024-01-15T11:25:16.277312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:43.332244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.277325+0000","last_change":"2024-01-15T11:25:15.235460+0000","last_active":"2024-01-15T11:25:16.277325+0000","last_peered":"2024-01-15T11:25:16.277325+0000","last_clean":"2024-01-15T11:25:16.277325+0000","last_became_active":"2024-01-15T11:25:15.235194+0000","last_became_peered":"2024-01-15T11:25:15.235194+0000","last_unstale":"2024-01-15T11:25:16.277325+0000","last_undegraded":"2024-01-15T11:25:16.277325+0000","last_fullsized":"2024-01-15T11:25:16.277325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:32.421454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.266744+0000","last_change":"2024-01-15T11:25:16.254349+0000","last_active":"2024-01-15T11:25:16.266744+0000","last_peered":"2024-01-15T11:25:16.266744+0000","last_clean":"2024-01-15T11:25:16.266744+0000","last_became_active":"2024-01-15T11:25:16.254068+0000","last_became_peered":"2024-01-15T11:25:16.254068+0000","last_unstale":"2024-01-15T11:25:16.266744+0000","last_undegraded":"2024-01-15T11:25:16.266744+0000","last_fullsized":"2024-01-15T11:25:16.266744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:56:33.010927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275958+0000","last_change":"2024-01-15T11:25:15.236870+0000","last_active":"2024-01-15T11:25:16.275958+0000","last_peered":"2024-01-15T11:25:16.275958+0000","last_clean":"2024-01-15T11:25:16.275958+0000","last_became_active":"2024-01-15T11:25:15.236597+0000","last_became_peered":"2024-01-15T11:25:15.236597+0000","last_unstale":"2024-01-15T11:25:16.275958+0000","last_undegraded":"2024-01-15T11:25:16.275958+0000","last_fullsized":"2024-01-15T11:25:16.275958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:04.223510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260775+0000","last_change":"2024-01-15T11:25:16.257709+0000","last_active":"2024-01-15T11:25:16.260775+0000","last_peered":"2024-01-15T11:25:16.260775+0000","last_clean":"2024-01-15T11:25:16.260775+0000","last_became_active":"2024-01-15T11:25:16.257415+0000","last_became_peered":"2024-01-15T11:25:16.257415+0000","last_unstale":"2024-01-15T11:25:16.260775+0000","last_undegraded":"2024-01-15T11:25:16.260775+0000","last_fullsized":"2024-01-15T11:25:16.260775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:43.332244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.259168+0000","last_change":"2024-01-15T11:25:15.233904+0000","last_active":"2024-01-15T11:25:16.259168+0000","last_peered":"2024-01-15T11:25:16.259168+0000","last_clean":"2024-01-15T11:25:16.259168+0000","last_became_active":"2024-01-15T11:25:15.232303+0000","last_became_peered":"2024-01-15T11:25:15.232303+0000","last_unstale":"2024-01-15T11:25:16.259168+0000","last_undegraded":"2024-01-15T11:25:16.259168+0000","last_fullsized":"2024-01-15T11:25:16.259168+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:14.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.271334+0000","last_change":"2024-01-15T11:25:16.265440+0000","last_active":"2024-01-15T11:25:16.271334+0000","last_peered":"2024-01-15T11:25:16.271334+0000","last_clean":"2024-01-15T11:25:16.271334+0000","last_became_active":"2024-01-15T11:25:16.265124+0000","last_became_peered":"2024-01-15T11:25:16.265124+0000","last_unstale":"2024-01-15T11:25:16.271334+0000","last_undegraded":"2024-01-15T11:25:16.271334+0000","last_fullsized":"2024-01-15T11:25:16.271334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:54.759984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.249605+0000","last_change":"2024-01-15T11:25:15.230668+0000","last_active":"2024-01-15T11:25:16.249605+0000","last_peered":"2024-01-15T11:25:16.249605+0000","last_clean":"2024-01-15T11:25:16.249605+0000","last_became_active":"2024-01-15T11:25:15.230323+0000","last_became_peered":"2024-01-15T11:25:15.230323+0000","last_unstale":"2024-01-15T11:25:16.249605+0000","last_undegraded":"2024-01-15T11:25:16.249605+0000","last_fullsized":"2024-01-15T11:25:16.249605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:25.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272584+0000","last_change":"2024-01-15T11:25:16.267799+0000","last_active":"2024-01-15T11:25:16.272584+0000","last_peered":"2024-01-15T11:25:16.272584+0000","last_clean":"2024-01-15T11:25:16.272584+0000","last_became_active":"2024-01-15T11:25:16.266612+0000","last_became_peered":"2024-01-15T11:25:16.266612+0000","last_unstale":"2024-01-15T11:25:16.272584+0000","last_undegraded":"2024-01-15T11:25:16.272584+0000","last_fullsized":"2024-01-15T11:25:16.272584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:17.504194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272359+0000","last_change":"2024-01-15T11:25:15.236904+0000","last_active":"2024-01-15T11:25:16.272359+0000","last_peered":"2024-01-15T11:25:16.272359+0000","last_clean":"2024-01-15T11:25:16.272359+0000","last_became_active":"2024-01-15T11:25:15.236628+0000","last_became_peered":"2024-01-15T11:25:15.236628+0000","last_unstale":"2024-01-15T11:25:16.272359+0000","last_undegraded":"2024-01-15T11:25:16.272359+0000","last_fullsized":"2024-01-15T11:25:16.272359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:42.327446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275356+0000","last_change":"2024-01-15T11:25:16.271784+0000","last_active":"2024-01-15T11:25:16.275356+0000","last_peered":"2024-01-15T11:25:16.275356+0000","last_clean":"2024-01-15T11:25:16.275356+0000","last_became_active":"2024-01-15T11:25:16.271432+0000","last_became_peered":"2024-01-15T11:25:16.271432+0000","last_unstale":"2024-01-15T11:25:16.275356+0000","last_undegraded":"2024-01-15T11:25:16.275356+0000","last_fullsized":"2024-01-15T11:25:16.275356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:33.426251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275216+0000","last_change":"2024-01-15T11:25:15.238136+0000","last_active":"2024-01-15T11:25:16.275216+0000","last_peered":"2024-01-15T11:25:16.275216+0000","last_clean":"2024-01-15T11:25:16.275216+0000","last_became_active":"2024-01-15T11:25:15.237522+0000","last_became_peered":"2024-01-15T11:25:15.237522+0000","last_unstale":"2024-01-15T11:25:16.275216+0000","last_undegraded":"2024-01-15T11:25:16.275216+0000","last_fullsized":"2024-01-15T11:25:16.275216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:16.499396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.270209+0000","last_change":"2024-01-15T11:25:16.263503+0000","last_active":"2024-01-15T11:25:16.270209+0000","last_peered":"2024-01-15T11:25:16.270209+0000","last_clean":"2024-01-15T11:25:16.270209+0000","last_became_active":"2024-01-15T11:25:16.263208+0000","last_became_peered":"2024-01-15T11:25:16.263208+0000","last_unstale":"2024-01-15T11:25:16.270209+0000","last_undegraded":"2024-01-15T11:25:16.270209+0000","last_fullsized":"2024-01-15T11:25:16.270209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:26.801860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.248350+0000","last_change":"2024-01-15T11:25:15.229209+0000","last_active":"2024-01-15T11:25:16.248350+0000","last_peered":"2024-01-15T11:25:16.248350+0000","last_clean":"2024-01-15T11:25:16.248350+0000","last_became_active":"2024-01-15T11:25:15.228899+0000","last_became_peered":"2024-01-15T11:25:15.228899+0000","last_unstale":"2024-01-15T11:25:16.248350+0000","last_undegraded":"2024-01-15T11:25:16.248350+0000","last_fullsized":"2024-01-15T11:25:16.248350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:56:32.006130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.266818+0000","last_change":"2024-01-15T11:25:16.259973+0000","last_active":"2024-01-15T11:25:16.266818+0000","last_peered":"2024-01-15T11:25:16.266818+0000","last_clean":"2024-01-15T11:25:16.266818+0000","last_became_active":"2024-01-15T11:25:16.259589+0000","last_became_peered":"2024-01-15T11:25:16.259589+0000","last_unstale":"2024-01-15T11:25:16.266818+0000","last_undegraded":"2024-01-15T11:25:16.266818+0000","last_fullsized":"2024-01-15T11:25:16.266818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.431909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.258986+0000","last_change":"2024-01-15T11:25:15.234071+0000","last_active":"2024-01-15T11:25:16.258986+0000","last_peered":"2024-01-15T11:25:16.258986+0000","last_clean":"2024-01-15T11:25:16.258986+0000","last_became_active":"2024-01-15T11:25:15.233460+0000","last_became_peered":"2024-01-15T11:25:15.233460+0000","last_unstale":"2024-01-15T11:25:16.258986+0000","last_undegraded":"2024-01-15T11:25:16.258986+0000","last_fullsized":"2024-01-15T11:25:16.258986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:30.739157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275800+0000","last_change":"2024-01-15T11:25:16.268001+0000","last_active":"2024-01-15T11:25:16.275800+0000","last_peered":"2024-01-15T11:25:16.275800+0000","last_clean":"2024-01-15T11:25:16.275800+0000","last_became_active":"2024-01-15T11:25:16.267277+0000","last_became_peered":"2024-01-15T11:25:16.267277+0000","last_unstale":"2024-01-15T11:25:16.275800+0000","last_undegraded":"2024-01-15T11:25:16.275800+0000","last_fullsized":"2024-01-15T11:25:16.275800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:59.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260263+0000","last_change":"2024-01-15T11:25:16.258019+0000","last_active":"2024-01-15T11:25:16.260263+0000","last_peered":"2024-01-15T11:25:16.260263+0000","last_clean":"2024-01-15T11:25:16.260263+0000","last_became_active":"2024-01-15T11:25:16.257755+0000","last_became_peered":"2024-01-15T11:25:16.257755+0000","last_unstale":"2024-01-15T11:25:16.260263+0000","last_undegraded":"2024-01-15T11:25:16.260263+0000","last_fullsized":"2024-01-15T11:25:16.260263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:05.228308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.243043+0000","last_change":"2024-01-15T11:25:15.235627+0000","last_active":"2024-01-15T11:25:16.243043+0000","last_peered":"2024-01-15T11:25:16.243043+0000","last_clean":"2024-01-15T11:25:16.243043+0000","last_became_active":"2024-01-15T11:25:15.235308+0000","last_became_peered":"2024-01-15T11:25:15.235308+0000","last_unstale":"2024-01-15T11:25:16.243043+0000","last_undegraded":"2024-01-15T11:25:16.243043+0000","last_fullsized":"2024-01-15T11:25:16.243043+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:58.480006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.261119+0000","last_change":"2024-01-15T11:25:16.260147+0000","last_active":"2024-01-15T11:25:16.261119+0000","last_peered":"2024-01-15T11:25:16.261119+0000","last_clean":"2024-01-15T11:25:16.261119+0000","last_became_active":"2024-01-15T11:25:16.259836+0000","last_became_peered":"2024-01-15T11:25:16.259836+0000","last_unstale":"2024-01-15T11:25:16.261119+0000","last_undegraded":"2024-01-15T11:25:16.261119+0000","last_fullsized":"2024-01-15T11:25:16.261119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:59.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.250071+0000","last_change":"2024-01-15T11:25:15.233565+0000","last_active":"2024-01-15T11:25:16.250071+0000","last_peered":"2024-01-15T11:25:16.250071+0000","last_clean":"2024-01-15T11:25:16.250071+0000","last_became_active":"2024-01-15T11:25:15.231919+0000","last_became_peered":"2024-01-15T11:25:15.231919+0000","last_unstale":"2024-01-15T11:25:16.250071+0000","last_undegraded":"2024-01-15T11:25:16.250071+0000","last_fullsized":"2024-01-15T11:25:16.250071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-15T11:25:15.227419+0000","last_change":"2024-01-15T11:25:03.506154+0000","last_active":"2024-01-15T11:25:15.227419+0000","last_peered":"2024-01-15T11:25:15.227419+0000","last_clean":"2024-01-15T11:25:15.227419+0000","last_became_active":"2024-01-15T11:25:03.504996+0000","last_became_peered":"2024-01-15T11:25:03.504996+0000","last_unstale":"2024-01-15T11:25:15.227419+0000","last_undegraded":"2024-01-15T11:25:15.227419+0000","last_fullsized":"2024-01-15T11:25:15.227419+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:02.494136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:02.494136+0000","last_clean_scrub_stamp":"2024-01-15T11:25:02.494136+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:22:33.043449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.271694+0000","last_change":"2024-01-15T11:25:16.270301+0000","last_active":"2024-01-15T11:25:16.271694+0000","last_peered":"2024-01-15T11:25:16.271694+0000","last_clean":"2024-01-15T11:25:16.271694+0000","last_became_active":"2024-01-15T11:25:16.269969+0000","last_became_peered":"2024-01-15T11:25:16.269969+0000","last_unstale":"2024-01-15T11:25:16.271694+0000","last_undegraded":"2024-01-15T11:25:16.271694+0000","last_fullsized":"2024-01-15T11:25:16.271694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.243429+0000","last_change":"2024-01-15T11:25:15.233382+0000","last_active":"2024-01-15T11:25:16.243429+0000","last_peered":"2024-01-15T11:25:16.243429+0000","last_clean":"2024-01-15T11:25:16.243429+0000","last_became_active":"2024-01-15T11:25:15.233111+0000","last_became_peered":"2024-01-15T11:25:15.233111+0000","last_unstale":"2024-01-15T11:25:16.243429+0000","last_undegraded":"2024-01-15T11:25:16.243429+0000","last_fullsized":"2024-01-15T11:25:16.243429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:50.865963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.244625+0000","last_change":"2024-01-15T11:25:15.237505+0000","last_active":"2024-01-15T11:25:16.244625+0000","last_peered":"2024-01-15T11:25:16.244625+0000","last_clean":"2024-01-15T11:25:16.244625+0000","last_became_active":"2024-01-15T11:25:15.237206+0000","last_became_peered":"2024-01-15T11:25:15.237206+0000","last_unstale":"2024-01-15T11:25:16.244625+0000","last_undegraded":"2024-01-15T11:25:16.244625+0000","last_fullsized":"2024-01-15T11:25:16.244625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:14.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.258881+0000","last_change":"2024-01-15T11:25:16.254087+0000","last_active":"2024-01-15T11:25:16.258881+0000","last_peered":"2024-01-15T11:25:16.258881+0000","last_clean":"2024-01-15T11:25:16.258881+0000","last_became_active":"2024-01-15T11:25:16.253790+0000","last_became_peered":"2024-01-15T11:25:16.253790+0000","last_unstale":"2024-01-15T11:25:16.258881+0000","last_undegraded":"2024-01-15T11:25:16.258881+0000","last_fullsized":"2024-01-15T11:25:16.258881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.247853+0000","last_change":"2024-01-15T11:25:15.240170+0000","last_active":"2024-01-15T11:25:16.247853+0000","last_peered":"2024-01-15T11:25:16.247853+0000","last_clean":"2024-01-15T11:25:16.247853+0000","last_became_active":"2024-01-15T11:25:15.239842+0000","last_became_peered":"2024-01-15T11:25:15.239842+0000","last_unstale":"2024-01-15T11:25:16.247853+0000","last_undegraded":"2024-01-15T11:25:16.247853+0000","last_fullsized":"2024-01-15T11:25:16.247853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:25.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.255085+0000","last_change":"2024-01-15T11:25:16.247896+0000","last_active":"2024-01-15T11:25:16.255085+0000","last_peered":"2024-01-15T11:25:16.255085+0000","last_clean":"2024-01-15T11:25:16.255085+0000","last_became_active":"2024-01-15T11:25:16.247606+0000","last_became_peered":"2024-01-15T11:25:16.247606+0000","last_unstale":"2024-01-15T11:25:16.255085+0000","last_undegraded":"2024-01-15T11:25:16.255085+0000","last_fullsized":"2024-01-15T11:25:16.255085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272739+0000","last_change":"2024-01-15T11:25:15.233571+0000","last_active":"2024-01-15T11:25:16.272739+0000","last_peered":"2024-01-15T11:25:16.272739+0000","last_clean":"2024-01-15T11:25:16.272739+0000","last_became_active":"2024-01-15T11:25:15.233180+0000","last_became_peered":"2024-01-15T11:25:15.233180+0000","last_unstale":"2024-01-15T11:25:16.272739+0000","last_undegraded":"2024-01-15T11:25:16.272739+0000","last_fullsized":"2024-01-15T11:25:16.272739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:53.755186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.258833+0000","last_change":"2024-01-15T11:25:16.253976+0000","last_active":"2024-01-15T11:25:16.258833+0000","last_peered":"2024-01-15T11:25:16.258833+0000","last_clean":"2024-01-15T11:25:16.258833+0000","last_became_active":"2024-01-15T11:25:16.253600+0000","last_became_peered":"2024-01-15T11:25:16.253600+0000","last_unstale":"2024-01-15T11:25:16.258833+0000","last_undegraded":"2024-01-15T11:25:16.258833+0000","last_fullsized":"2024-01-15T11:25:16.258833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:17.504194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.243339+0000","last_change":"2024-01-15T11:25:15.235253+0000","last_active":"2024-01-15T11:25:16.243339+0000","last_peered":"2024-01-15T11:25:16.243339+0000","last_clean":"2024-01-15T11:25:16.243339+0000","last_became_active":"2024-01-15T11:25:15.234844+0000","last_became_peered":"2024-01-15T11:25:15.234844+0000","last_unstale":"2024-01-15T11:25:16.243339+0000","last_undegraded":"2024-01-15T11:25:16.243339+0000","last_fullsized":"2024-01-15T11:25:16.243339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:18.628480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.264574+0000","last_change":"2024-01-15T11:25:16.257934+0000","last_active":"2024-01-15T11:25:16.264574+0000","last_peered":"2024-01-15T11:25:16.264574+0000","last_clean":"2024-01-15T11:25:16.264574+0000","last_became_active":"2024-01-15T11:25:16.257566+0000","last_became_peered":"2024-01-15T11:25:16.257566+0000","last_unstale":"2024-01-15T11:25:16.264574+0000","last_undegraded":"2024-01-15T11:25:16.264574+0000","last_fullsized":"2024-01-15T11:25:16.264574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:23.083850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.249457+0000","last_change":"2024-01-15T11:25:15.237581+0000","last_active":"2024-01-15T11:25:16.249457+0000","last_peered":"2024-01-15T11:25:16.249457+0000","last_clean":"2024-01-15T11:25:16.249457+0000","last_became_active":"2024-01-15T11:25:15.237319+0000","last_became_peered":"2024-01-15T11:25:15.237319+0000","last_unstale":"2024-01-15T11:25:16.249457+0000","last_undegraded":"2024-01-15T11:25:16.249457+0000","last_fullsized":"2024-01-15T11:25:16.249457+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260508+0000","last_change":"2024-01-15T11:25:16.256476+0000","last_active":"2024-01-15T11:25:16.260508+0000","last_peered":"2024-01-15T11:25:16.260508+0000","last_clean":"2024-01-15T11:25:16.260508+0000","last_became_active":"2024-01-15T11:25:16.256198+0000","last_became_peered":"2024-01-15T11:25:16.256198+0000","last_unstale":"2024-01-15T11:25:16.260508+0000","last_undegraded":"2024-01-15T11:25:16.260508+0000","last_fullsized":"2024-01-15T11:25:16.260508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.248235+0000","last_change":"2024-01-15T11:25:15.235154+0000","last_active":"2024-01-15T11:25:16.248235+0000","last_peered":"2024-01-15T11:25:16.248235+0000","last_clean":"2024-01-15T11:25:16.248235+0000","last_became_active":"2024-01-15T11:25:15.234870+0000","last_became_peered":"2024-01-15T11:25:15.234870+0000","last_unstale":"2024-01-15T11:25:16.248235+0000","last_undegraded":"2024-01-15T11:25:16.248235+0000","last_fullsized":"2024-01-15T11:25:16.248235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:53.755186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260309+0000","last_change":"2024-01-15T11:25:16.254780+0000","last_active":"2024-01-15T11:25:16.260309+0000","last_peered":"2024-01-15T11:25:16.260309+0000","last_clean":"2024-01-15T11:25:16.260309+0000","last_became_active":"2024-01-15T11:25:16.254519+0000","last_became_peered":"2024-01-15T11:25:16.254519+0000","last_unstale":"2024-01-15T11:25:16.260309+0000","last_undegraded":"2024-01-15T11:25:16.260309+0000","last_fullsized":"2024-01-15T11:25:16.260309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:17.504194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.277415+0000","last_change":"2024-01-15T11:25:15.238538+0000","last_active":"2024-01-15T11:25:16.277415+0000","last_peered":"2024-01-15T11:25:16.277415+0000","last_clean":"2024-01-15T11:25:16.277415+0000","last_became_active":"2024-01-15T11:25:15.238222+0000","last_became_peered":"2024-01-15T11:25:15.238222+0000","last_unstale":"2024-01-15T11:25:16.277415+0000","last_undegraded":"2024-01-15T11:25:16.277415+0000","last_fullsized":"2024-01-15T11:25:16.277415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:30.739157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272839+0000","last_change":"2024-01-15T11:25:16.262608+0000","last_active":"2024-01-15T11:25:16.272839+0000","last_peered":"2024-01-15T11:25:16.272839+0000","last_clean":"2024-01-15T11:25:16.272839+0000","last_became_active":"2024-01-15T11:25:16.262318+0000","last_became_peered":"2024-01-15T11:25:16.262318+0000","last_unstale":"2024-01-15T11:25:16.272839+0000","last_undegraded":"2024-01-15T11:25:16.272839+0000","last_fullsized":"2024-01-15T11:25:16.272839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:59.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.242958+0000","last_change":"2024-01-15T11:25:15.232704+0000","last_active":"2024-01-15T11:25:16.242958+0000","last_peered":"2024-01-15T11:25:16.242958+0000","last_clean":"2024-01-15T11:25:16.242958+0000","last_became_active":"2024-01-15T11:25:15.232191+0000","last_became_peered":"2024-01-15T11:25:15.232191+0000","last_unstale":"2024-01-15T11:25:16.242958+0000","last_undegraded":"2024-01-15T11:25:16.242958+0000","last_fullsized":"2024-01-15T11:25:16.242958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:16.499396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.269902+0000","last_change":"2024-01-15T11:25:16.260276+0000","last_active":"2024-01-15T11:25:16.269902+0000","last_peered":"2024-01-15T11:25:16.269902+0000","last_clean":"2024-01-15T11:25:16.269902+0000","last_became_active":"2024-01-15T11:25:16.259962+0000","last_became_peered":"2024-01-15T11:25:16.259962+0000","last_unstale":"2024-01-15T11:25:16.269902+0000","last_undegraded":"2024-01-15T11:25:16.269902+0000","last_fullsized":"2024-01-15T11:25:16.269902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:08:06.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275885+0000","last_change":"2024-01-15T11:25:15.230446+0000","last_active":"2024-01-15T11:25:16.275885+0000","last_peered":"2024-01-15T11:25:16.275885+0000","last_clean":"2024-01-15T11:25:16.275885+0000","last_became_active":"2024-01-15T11:25:15.230176+0000","last_became_peered":"2024-01-15T11:25:15.230176+0000","last_unstale":"2024-01-15T11:25:16.275885+0000","last_undegraded":"2024-01-15T11:25:16.275885+0000","last_fullsized":"2024-01-15T11:25:16.275885+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:32.421454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.268751+0000","last_change":"2024-01-15T11:25:16.262438+0000","last_active":"2024-01-15T11:25:16.268751+0000","last_peered":"2024-01-15T11:25:16.268751+0000","last_clean":"2024-01-15T11:25:16.268751+0000","last_became_active":"2024-01-15T11:25:16.261849+0000","last_became_peered":"2024-01-15T11:25:16.261849+0000","last_unstale":"2024-01-15T11:25:16.268751+0000","last_undegraded":"2024-01-15T11:25:16.268751+0000","last_fullsized":"2024-01-15T11:25:16.268751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:26.586428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.271673+0000","last_change":"2024-01-15T11:25:15.235007+0000","last_active":"2024-01-15T11:25:16.271673+0000","last_peered":"2024-01-15T11:25:16.271673+0000","last_clean":"2024-01-15T11:25:16.271673+0000","last_became_active":"2024-01-15T11:25:15.234709+0000","last_became_peered":"2024-01-15T11:25:15.234709+0000","last_unstale":"2024-01-15T11:25:16.271673+0000","last_undegraded":"2024-01-15T11:25:16.271673+0000","last_fullsized":"2024-01-15T11:25:16.271673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:25.427111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.270455+0000","last_change":"2024-01-15T11:25:16.263407+0000","last_active":"2024-01-15T11:25:16.270455+0000","last_peered":"2024-01-15T11:25:16.270455+0000","last_clean":"2024-01-15T11:25:16.270455+0000","last_became_active":"2024-01-15T11:25:16.263120+0000","last_became_peered":"2024-01-15T11:25:16.263120+0000","last_unstale":"2024-01-15T11:25:16.270455+0000","last_undegraded":"2024-01-15T11:25:16.270455+0000","last_fullsized":"2024-01-15T11:25:16.270455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:28:18.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272390+0000","last_change":"2024-01-15T11:25:16.258910+0000","last_active":"2024-01-15T11:25:16.272390+0000","last_peered":"2024-01-15T11:25:16.272390+0000","last_clean":"2024-01-15T11:25:16.272390+0000","last_became_active":"2024-01-15T11:25:16.258619+0000","last_became_peered":"2024-01-15T11:25:16.258619+0000","last_unstale":"2024-01-15T11:25:16.272390+0000","last_undegraded":"2024-01-15T11:25:16.272390+0000","last_fullsized":"2024-01-15T11:25:16.272390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.263680+0000","last_change":"2024-01-15T11:25:16.259468+0000","last_active":"2024-01-15T11:25:16.263680+0000","last_peered":"2024-01-15T11:25:16.263680+0000","last_clean":"2024-01-15T11:25:16.263680+0000","last_became_active":"2024-01-15T11:25:16.258703+0000","last_became_peered":"2024-01-15T11:25:16.258703+0000","last_unstale":"2024-01-15T11:25:16.263680+0000","last_undegraded":"2024-01-15T11:25:16.263680+0000","last_fullsized":"2024-01-15T11:25:16.263680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.273292+0000","last_change":"2024-01-15T11:25:16.266445+0000","last_active":"2024-01-15T11:25:16.273292+0000","last_peered":"2024-01-15T11:25:16.273292+0000","last_clean":"2024-01-15T11:25:16.273292+0000","last_became_active":"2024-01-15T11:25:16.265486+0000","last_became_peered":"2024-01-15T11:25:16.265486+0000","last_unstale":"2024-01-15T11:25:16.273292+0000","last_undegraded":"2024-01-15T11:25:16.273292+0000","last_fullsized":"2024-01-15T11:25:16.273292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:17:27.965755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.261004+0000","last_change":"2024-01-15T11:25:16.259210+0000","last_active":"2024-01-15T11:25:16.261004+0000","last_peered":"2024-01-15T11:25:16.261004+0000","last_clean":"2024-01-15T11:25:16.261004+0000","last_became_active":"2024-01-15T11:25:16.258919+0000","last_became_peered":"2024-01-15T11:25:16.258919+0000","last_unstale":"2024-01-15T11:25:16.261004+0000","last_undegraded":"2024-01-15T11:25:16.261004+0000","last_fullsized":"2024-01-15T11:25:16.261004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:59.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.266744+0000","last_change":"2024-01-15T11:25:16.264604+0000","last_active":"2024-01-15T11:25:16.266744+0000","last_peered":"2024-01-15T11:25:16.266744+0000","last_clean":"2024-01-15T11:25:16.266744+0000","last_became_active":"2024-01-15T11:25:16.263950+0000","last_became_peered":"2024-01-15T11:25:16.263950+0000","last_unstale":"2024-01-15T11:25:16.266744+0000","last_undegraded":"2024-01-15T11:25:16.266744+0000","last_fullsized":"2024-01-15T11:25:16.266744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:43.332244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.260084+0000","last_change":"2024-01-15T11:25:16.256747+0000","last_active":"2024-01-15T11:25:16.260084+0000","last_peered":"2024-01-15T11:25:16.260084+0000","last_clean":"2024-01-15T11:25:16.260084+0000","last_became_active":"2024-01-15T11:25:16.256464+0000","last_became_peered":"2024-01-15T11:25:16.256464+0000","last_unstale":"2024-01-15T11:25:16.260084+0000","last_undegraded":"2024-01-15T11:25:16.260084+0000","last_fullsized":"2024-01-15T11:25:16.260084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272891+0000","last_change":"2024-01-15T11:25:16.268099+0000","last_active":"2024-01-15T11:25:16.272891+0000","last_peered":"2024-01-15T11:25:16.272891+0000","last_clean":"2024-01-15T11:25:16.272891+0000","last_became_active":"2024-01-15T11:25:16.267826+0000","last_became_peered":"2024-01-15T11:25:16.267826+0000","last_unstale":"2024-01-15T11:25:16.272891+0000","last_undegraded":"2024-01-15T11:25:16.272891+0000","last_fullsized":"2024-01-15T11:25:16.272891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.265130+0000","last_change":"2024-01-15T11:25:16.263641+0000","last_active":"2024-01-15T11:25:16.265130+0000","last_peered":"2024-01-15T11:25:16.265130+0000","last_clean":"2024-01-15T11:25:16.265130+0000","last_became_active":"2024-01-15T11:25:16.263368+0000","last_became_peered":"2024-01-15T11:25:16.263368+0000","last_unstale":"2024-01-15T11:25:16.265130+0000","last_undegraded":"2024-01-15T11:25:16.265130+0000","last_fullsized":"2024-01-15T11:25:16.265130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:05.228308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.268937+0000","last_change":"2024-01-15T11:25:16.262170+0000","last_active":"2024-01-15T11:25:16.268937+0000","last_peered":"2024-01-15T11:25:16.268937+0000","last_clean":"2024-01-15T11:25:16.268937+0000","last_became_active":"2024-01-15T11:25:16.261695+0000","last_became_peered":"2024-01-15T11:25:16.261695+0000","last_unstale":"2024-01-15T11:25:16.268937+0000","last_undegraded":"2024-01-15T11:25:16.268937+0000","last_fullsized":"2024-01-15T11:25:16.268937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.267627+0000","last_change":"2024-01-15T11:25:16.265406+0000","last_active":"2024-01-15T11:25:16.267627+0000","last_peered":"2024-01-15T11:25:16.267627+0000","last_clean":"2024-01-15T11:25:16.267627+0000","last_became_active":"2024-01-15T11:25:16.265150+0000","last_became_peered":"2024-01-15T11:25:16.265150+0000","last_unstale":"2024-01-15T11:25:16.267627+0000","last_undegraded":"2024-01-15T11:25:16.267627+0000","last_fullsized":"2024-01-15T11:25:16.267627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:19:46.378784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.275073+0000","last_change":"2024-01-15T11:25:16.271625+0000","last_active":"2024-01-15T11:25:16.275073+0000","last_peered":"2024-01-15T11:25:16.275073+0000","last_clean":"2024-01-15T11:25:16.275073+0000","last_became_active":"2024-01-15T11:25:16.271361+0000","last_became_peered":"2024-01-15T11:25:16.271361+0000","last_unstale":"2024-01-15T11:25:16.275073+0000","last_undegraded":"2024-01-15T11:25:16.275073+0000","last_fullsized":"2024-01-15T11:25:16.275073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:02:52.680406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272252+0000","last_change":"2024-01-15T11:25:16.271113+0000","last_active":"2024-01-15T11:25:16.272252+0000","last_peered":"2024-01-15T11:25:16.272252+0000","last_clean":"2024-01-15T11:25:16.272252+0000","last_became_active":"2024-01-15T11:25:16.270795+0000","last_became_peered":"2024-01-15T11:25:16.270795+0000","last_unstale":"2024-01-15T11:25:16.272252+0000","last_undegraded":"2024-01-15T11:25:16.272252+0000","last_fullsized":"2024-01-15T11:25:16.272252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.431909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.272566+0000","last_change":"2024-01-15T11:25:16.262518+0000","last_active":"2024-01-15T11:25:16.272566+0000","last_peered":"2024-01-15T11:25:16.272566+0000","last_clean":"2024-01-15T11:25:16.272566+0000","last_became_active":"2024-01-15T11:25:16.262049+0000","last_became_peered":"2024-01-15T11:25:16.262049+0000","last_unstale":"2024-01-15T11:25:16.272566+0000","last_undegraded":"2024-01-15T11:25:16.272566+0000","last_fullsized":"2024-01-15T11:25:16.272566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.431909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":66222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":64770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":774144,"data_stored":648203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":66222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":64736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":64700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T11:25:20.732 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T11:25:20.732 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-15T11:25:20.732 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-15T11:25:20.733 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-15T11:25:20.733 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-15T11:25:20.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.890+0000 7f59bc5a4700 1 Processor -- start 2024-01-15T11:25:20.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.890+0000 7f59bc5a4700 1 -- start start 2024-01-15T11:25:20.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.891+0000 7f59bc5a4700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b415af10 0x7f59b415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.891+0000 7f59bc5a4700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f59b40ce140 0x7f59b415ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.891+0000 7f59bc5a4700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b41414c0 con 0x7f59b415f460 2024-01-15T11:25:20.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.891+0000 7f59bc5a4700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b415f8c0 con 0x7f59b40ce140 2024-01-15T11:25:20.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.891+0000 7f59bc5a4700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b40ce9f0 con 0x7f59b415af10 2024-01-15T11:25:20.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59bab41700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f59b415f460 0x7f59b415b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:37456/0 (socket says 172.21.15.113:37456) 2024-01-15T11:25:20.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59bab41700 1 -- 172.21.15.113:0/928403293 learned_addr learned my addr 172.21.15.113:0/928403293 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:20.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59ba340700 1 --2- 172.21.15.113:0/928403293 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b415af10 0x7f59b415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147068755 0 0) 0x7f59b41414c0 con 0x7f59b415f460 2024-01-15T11:25:20.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b9b3f700 1 --2- 172.21.15.113:0/928403293 >> v2:172.21.15.141:3300/0 conn(0x7f59b40ce140 0x7f59b415ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a00035b0 con 0x7f59b415f460 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1770359915 0 0) 0x7f59a00035b0 con 0x7f59b415f460 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b415af10 msgr2=0x7f59b415e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 --2- 172.21.15.113:0/928403293 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b415af10 0x7f59b415e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 >> v2:172.21.15.141:3300/0 conn(0x7f59b40ce140 msgr2=0x7f59b415ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 --2- 172.21.15.113:0/928403293 >> v2:172.21.15.141:3300/0 conn(0x7f59b40ce140 0x7f59b415ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b40cecd0 con 0x7f59b415f460 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f59ac0019f0 con 0x7f59b415f460 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 >> v1:172.21.15.113:6789/0 conn(0x7f59b415f460 legacy=0x7f59b415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 --2- 172.21.15.113:0/928403293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59a0003a00 0x7f59a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 --2- 172.21.15.113:0/928403293 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b415af10 0x7f59a0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 --2- 172.21.15.113:0/928403293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a0004d90 0x7f59a0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b415f8c0 con 0x7f59a0004d90 2024-01-15T11:25:20.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b40ce9f0 con 0x7f59a0003a00 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a000dc20 con 0x7f59b415af10 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59bab41700 1 --2- 172.21.15.113:0/928403293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a0004d90 0x7f59a0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59b9b3f700 1 --2- 172.21.15.113:0/928403293 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b415af10 0x7f59a0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59ba340700 1 --2- 172.21.15.113:0/928403293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59a0003a00 0x7f59a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59bab41700 1 -- 172.21.15.113:0/928403293 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b415af10 msgr2=0x7f59a0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59bab41700 1 --2- 172.21.15.113:0/928403293 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b415af10 0x7f59a0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59bab41700 1 -- 172.21.15.113:0/928403293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59a0003a00 msgr2=0x7f59a0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59bab41700 1 --2- 172.21.15.113:0/928403293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59a0003a00 0x7f59a0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.892+0000 7f59bab41700 1 -- 172.21.15.113:0/928403293 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a00035b0 con 0x7f59a0004d90 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.893+0000 7f59bab41700 1 --2- 172.21.15.113:0/928403293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a0004d90 0x7f59a0005210 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59ac00a8d0 tx=0x7f59a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=763618a85a431bb5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.893+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ac00a340 con 0x7f59a0004d90 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.893+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ac012600 con 0x7f59a0004d90 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.893+0000 7f59b933e700 1 -- 172.21.15.113:0/928403293 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0129b0 con 0x7f59a0004d90 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.893+0000 7f59bc5a4700 1 -- 172.21.15.113:0/928403293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a0004d90 msgr2=0x7f59a0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.893+0000 7f59bc5a4700 1 --2- 172.21.15.113:0/928403293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a0004d90 0x7f59a0005210 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59ac00a8d0 tx=0x7f59a0005880 comp rx=0 tx=0).stop 2024-01-15T11:25:20.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.893+0000 7f59bab41700 1 -- 172.21.15.113:0/928403293 reap_dead start 2024-01-15T11:25:20.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.893+0000 7f59bc5a4700 1 -- 172.21.15.113:0/928403293 shutdown_connections 2024-01-15T11:25:20.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.893+0000 7f59bc5a4700 1 -- 172.21.15.113:0/928403293 >> 172.21.15.113:0/928403293 conn(0x7f59b40cd570 msgr2=0x7f59b40cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:20.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.893+0000 7f59bc5a4700 1 -- 172.21.15.113:0/928403293 shutdown_connections 2024-01-15T11:25:20.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.893+0000 7f59bc5a4700 1 -- 172.21.15.113:0/928403293 wait complete. 2024-01-15T11:25:20.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.895+0000 7f59bc5a4700 1 Processor -- start 2024-01-15T11:25:20.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.895+0000 7f59bc5a4700 1 -- start start 2024-01-15T11:25:20.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.895+0000 7f59bc5a4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a0004d90 0x7f59b40a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.895+0000 7f59bc5a4700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b40ab4f0 0x7f59b413fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.895+0000 7f59bc5a4700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b4159240 0x7f59b40b1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.895+0000 7f59bc5a4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b40ab0d0 con 0x7f59a0004d90 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.895+0000 7f59bc5a4700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b41402c0 con 0x7f59b40ab4f0 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.895+0000 7f59bc5a4700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b40aa450 con 0x7f59b4159240 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.895+0000 7f59ba340700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a0004d90 0x7f59b40a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.895+0000 7f59ba340700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a0004d90 0x7f59b40a9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45516/0 (socket says 172.21.15.113:45516) 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.895+0000 7f59ba340700 1 -- 172.21.15.113:0/3389995183 learned_addr learned my addr 172.21.15.113:0/3389995183 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59b9b3f700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b40ab4f0 0x7f59b413fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59bab41700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b4159240 0x7f59b40b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59ba340700 1 -- 172.21.15.113:0/3389995183 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b4159240 msgr2=0x7f59b40b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59ba340700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b4159240 0x7f59b40b1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59ba340700 1 -- 172.21.15.113:0/3389995183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b40ab4f0 msgr2=0x7f59b413fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59ba340700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b40ab4f0 0x7f59b413fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:20.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59ba340700 1 -- 172.21.15.113:0/3389995183 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b40aa730 con 0x7f59a0004d90 2024-01-15T11:25:20.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59ba340700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a0004d90 0x7f59b40a9e70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f59a4011aa0 tx=0x7f59a4011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:20.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59aaffd700 1 -- 172.21.15.113:0/3389995183 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a4011ed0 con 0x7f59a0004d90 2024-01-15T11:25:20.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59aaffd700 1 -- 172.21.15.113:0/3389995183 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a400f940 con 0x7f59a0004d90 2024-01-15T11:25:20.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59bc5a4700 1 -- 172.21.15.113:0/3389995183 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b40aaa10 con 0x7f59a0004d90 2024-01-15T11:25:20.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59aaffd700 1 -- 172.21.15.113:0/3389995183 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a400fcf0 con 0x7f59a0004d90 2024-01-15T11:25:20.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.896+0000 7f59bc5a4700 1 -- 172.21.15.113:0/3389995183 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b41596f0 con 0x7f59a0004d90 2024-01-15T11:25:20.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.897+0000 7f59bc5a4700 1 -- 172.21.15.113:0/3389995183 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b4067f60 con 0x7f59a0004d90 2024-01-15T11:25:20.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.897+0000 7f59aaffd700 1 -- 172.21.15.113:0/3389995183 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f59a4014440 con 0x7f59a0004d90 2024-01-15T11:25:20.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.898+0000 7f59aaffd700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f599806b350 0x7f599806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:20.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.898+0000 7f59b9b3f700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f599806b350 0x7f599806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:20.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.898+0000 7f59aaffd700 1 -- 172.21.15.113:0/3389995183 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59a408a130 con 0x7f59a0004d90 2024-01-15T11:25:20.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.898+0000 7f59b9b3f700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f599806b350 0x7f599806d810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f59b4159ea0 tx=0x7f59b0009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:20.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:20.902+0000 7f59aaffd700 1 -- 172.21.15.113:0/3389995183 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59a40550f0 con 0x7f59a0004d90 2024-01-15T11:25:21.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.145+0000 7f59bc5a4700 1 -- 172.21.15.113:0/3389995183 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f59b4072e00 con 0x7f59a0004d90 2024-01-15T11:25:21.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.148+0000 7f59aaffd700 1 -- 172.21.15.113:0/3389995183 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171564 (secure 0 0 0) 0x7f59a4058710 con 0x7f59a0004d90 2024-01-15T11:25:21.149 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:21.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.151+0000 7f59bc5a4700 1 -- 172.21.15.113:0/3389995183 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f599806b350 msgr2=0x7f599806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.151+0000 7f59bc5a4700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f599806b350 0x7f599806d810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f59b4159ea0 tx=0x7f59b0009210 comp rx=0 tx=0).stop 2024-01-15T11:25:21.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.151+0000 7f59bc5a4700 1 -- 172.21.15.113:0/3389995183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a0004d90 msgr2=0x7f59b40a9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.151+0000 7f59bc5a4700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a0004d90 0x7f59b40a9e70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f59a4011aa0 tx=0x7f59a4011db0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.151+0000 7f59bc5a4700 1 -- 172.21.15.113:0/3389995183 shutdown_connections 2024-01-15T11:25:21.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.151+0000 7f59bc5a4700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f599806b350 0x7f599806d810 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.151+0000 7f59bc5a4700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f59b4159240 0x7f59b40b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.151+0000 7f59bc5a4700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a0004d90 0x7f59b40a9e70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.151+0000 7f59bc5a4700 1 --2- 172.21.15.113:0/3389995183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f59b40ab4f0 0x7f59b413fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.151+0000 7f59bc5a4700 1 -- 172.21.15.113:0/3389995183 >> 172.21.15.113:0/3389995183 conn(0x7f59b40cd570 msgr2=0x7f59b406b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:21.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.151+0000 7f59bc5a4700 1 -- 172.21.15.113:0/3389995183 shutdown_connections 2024-01-15T11:25:21.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.151+0000 7f59bc5a4700 1 -- 172.21.15.113:0/3389995183 wait complete. 2024-01-15T11:25:21.161 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":5,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3789331922},{"type":"v1","addr":"172.21.15.113:6833","nonce":3789331922}]},"active_addr":"172.21.15.113:6833/3789331922","active_change":"2024-01-15T11:24:56.100162+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.113:0","nonce":455545205}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.113:0","nonce":1892149967}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.113:0","nonce":4011072172}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.113:0","nonce":3332146779}]}]} 2024-01-15T11:25:21.164 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-15T11:25:21.164 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-15T11:25:21.165 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:21.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.287+0000 7f23d539b700 1 Processor -- start 2024-01-15T11:25:21.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.287+0000 7f23d539b700 1 -- start start 2024-01-15T11:25:21.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.288+0000 7f23d539b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d015e1e0 0x7f23d00d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:21.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.288+0000 7f23d539b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23d00d3a00 0x7f23d00d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:21.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.288+0000 7f23d539b700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d00d20b0 con 0x7f23d00d0db0 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.288+0000 7f23d539b700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d00d3150 con 0x7f23d00d3a00 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.288+0000 7f23d539b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00d1240 con 0x7f23d015e1e0 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.288+0000 7f23ce7fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23d00d3a00 0x7f23d00d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.288+0000 7f23ce7fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23d00d3a00 0x7f23d00d0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45520/0 (socket says 172.21.15.113:45520) 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.288+0000 7f23ce7fc700 1 -- 172.21.15.113:0/2947825987 learned_addr learned my addr 172.21.15.113:0/2947825987 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.288+0000 7f23ceffd700 1 --2- 172.21.15.113:0/2947825987 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d015e1e0 0x7f23d00d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.288+0000 7f23ce7fc700 1 -- 172.21.15.113:0/2947825987 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d015e1e0 msgr2=0x7f23d00d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.288+0000 7f23ce7fc700 1 --2- 172.21.15.113:0/2947825987 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d015e1e0 0x7f23d00d34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.288+0000 7f23ce7fc700 1 -- 172.21.15.113:0/2947825987 >> v1:172.21.15.141:6789/0 conn(0x7f23d00d0db0 legacy=0x7f23d006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23ce7fc700 1 -- 172.21.15.113:0/2947825987 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d00d2390 con 0x7f23d00d3a00 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23ce7fc700 1 --2- 172.21.15.113:0/2947825987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d00d3a00 0x7f23d00d0870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f23c00098f0 tx=0x7f23c0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3822525e4a12e7b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23cdffb700 1 -- 172.21.15.113:0/2947825987 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c000b8a0 con 0x7f23d00d3a00 2024-01-15T11:25:21.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23cdffb700 1 -- 172.21.15.113:0/2947825987 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c000ba00 con 0x7f23d00d3a00 2024-01-15T11:25:21.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23cdffb700 1 -- 172.21.15.113:0/2947825987 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c000bd90 con 0x7f23d00d3a00 2024-01-15T11:25:21.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23d539b700 1 -- 172.21.15.113:0/2947825987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d00d3a00 msgr2=0x7f23d00d0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23d539b700 1 --2- 172.21.15.113:0/2947825987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d00d3a00 0x7f23d00d0870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f23c00098f0 tx=0x7f23c0009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:21.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23d539b700 1 -- 172.21.15.113:0/2947825987 shutdown_connections 2024-01-15T11:25:21.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23d539b700 1 --2- 172.21.15.113:0/2947825987 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d015e1e0 0x7f23d00d34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23d539b700 1 --2- 172.21.15.113:0/2947825987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d00d3a00 0x7f23d00d0870 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23d539b700 1 -- 172.21.15.113:0/2947825987 >> 172.21.15.113:0/2947825987 conn(0x7f23d00b5b50 msgr2=0x7f23d00b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:21.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23d539b700 1 -- 172.21.15.113:0/2947825987 shutdown_connections 2024-01-15T11:25:21.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.289+0000 7f23d539b700 1 -- 172.21.15.113:0/2947825987 wait complete. 2024-01-15T11:25:21.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.291+0000 7f23d539b700 1 Processor -- start 2024-01-15T11:25:21.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.291+0000 7f23d539b700 1 -- start start 2024-01-15T11:25:21.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.291+0000 7f23d539b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d00d0db0 0x7f23d00c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:21.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.291+0000 7f23d539b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d00d3a00 0x7f23d01f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:21.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.291+0000 7f23d539b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d015e1e0 0x7f23d00cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:21.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.291+0000 7f23d539b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00cd040 con 0x7f23d00d0db0 2024-01-15T11:25:21.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.291+0000 7f23d539b700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00cc3c0 con 0x7f23d00d3a00 2024-01-15T11:25:21.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.291+0000 7f23d539b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00cc6a0 con 0x7f23d015e1e0 2024-01-15T11:25:21.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.291+0000 7f23ceffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d00d0db0 0x7f23d00c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:21.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23ceffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d00d0db0 0x7f23d00c9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45526/0 (socket says 172.21.15.113:45526) 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23ceffd700 1 -- 172.21.15.113:0/2725559699 learned_addr learned my addr 172.21.15.113:0/2725559699 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23cf7fe700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d015e1e0 0x7f23d00cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23ce7fc700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d00d3a00 0x7f23d01f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23ceffd700 1 -- 172.21.15.113:0/2725559699 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d015e1e0 msgr2=0x7f23d00cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23ceffd700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d015e1e0 0x7f23d00cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23ceffd700 1 -- 172.21.15.113:0/2725559699 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d00d3a00 msgr2=0x7f23d01f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23ceffd700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d00d3a00 0x7f23d01f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23ceffd700 1 -- 172.21.15.113:0/2725559699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c0009550 con 0x7f23d00d0db0 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23ce7fc700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d00d3a00 0x7f23d01f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23ceffd700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d00d0db0 0x7f23d00c9c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f23c400d800 tx=0x7f23c400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23b77fe700 1 -- 172.21.15.113:0/2725559699 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c40125f0 con 0x7f23d00d0db0 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23b77fe700 1 -- 172.21.15.113:0/2725559699 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c4012750 con 0x7f23d00d0db0 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23d539b700 1 -- 172.21.15.113:0/2725559699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d00cc980 con 0x7f23d00d0db0 2024-01-15T11:25:21.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23b77fe700 1 -- 172.21.15.113:0/2725559699 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c4012b00 con 0x7f23d00d0db0 2024-01-15T11:25:21.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.292+0000 7f23d539b700 1 -- 172.21.15.113:0/2725559699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d0152700 con 0x7f23d00d0db0 2024-01-15T11:25:21.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.293+0000 7f23d539b700 1 -- 172.21.15.113:0/2725559699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d00751e0 con 0x7f23d00d0db0 2024-01-15T11:25:21.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.293+0000 7f23b77fe700 1 -- 172.21.15.113:0/2725559699 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f23c4012c60 con 0x7f23d00d0db0 2024-01-15T11:25:21.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.294+0000 7f23b77fe700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f23bc06b350 0x7f23bc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:21.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.294+0000 7f23b77fe700 1 -- 172.21.15.113:0/2725559699 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23c4089e10 con 0x7f23d00d0db0 2024-01-15T11:25:21.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.294+0000 7f23ce7fc700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f23bc06b350 0x7f23bc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:21.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.294+0000 7f23ce7fc700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f23bc06b350 0x7f23bc06d810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f23c0000c00 tx=0x7f23c0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:21.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.298+0000 7f23b77fe700 1 -- 172.21.15.113:0/2725559699 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23c4054d10 con 0x7f23d00d0db0 2024-01-15T11:25:21.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.489+0000 7f23d539b700 1 -- 172.21.15.113:0/2725559699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23d00c9050 con 0x7f23d00d0db0 2024-01-15T11:25:21.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.490+0000 7f23b77fe700 1 -- 172.21.15.113:0/2725559699 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f23c4058330 con 0x7f23d00d0db0 2024-01-15T11:25:21.489 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:21.489 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:21.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.492+0000 7f23d539b700 1 -- 172.21.15.113:0/2725559699 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f23bc06b350 msgr2=0x7f23bc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.492+0000 7f23d539b700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f23bc06b350 0x7f23bc06d810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f23c0000c00 tx=0x7f23c0008030 comp rx=0 tx=0).stop 2024-01-15T11:25:21.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.492+0000 7f23d539b700 1 -- 172.21.15.113:0/2725559699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d00d0db0 msgr2=0x7f23d00c9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.492+0000 7f23d539b700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d00d0db0 0x7f23d00c9c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f23c400d800 tx=0x7f23c400db10 comp rx=0 tx=0).stop 2024-01-15T11:25:21.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.492+0000 7f23d539b700 1 -- 172.21.15.113:0/2725559699 shutdown_connections 2024-01-15T11:25:21.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.492+0000 7f23d539b700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f23bc06b350 0x7f23bc06d810 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.492+0000 7f23d539b700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d015e1e0 0x7f23d00cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.492+0000 7f23d539b700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d00d0db0 0x7f23d00c9c80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.492+0000 7f23d539b700 1 --2- 172.21.15.113:0/2725559699 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d00d3a00 0x7f23d01f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.492+0000 7f23d539b700 1 -- 172.21.15.113:0/2725559699 >> 172.21.15.113:0/2725559699 conn(0x7f23d00b5b50 msgr2=0x7f23d00ce490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:21.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.492+0000 7f23d539b700 1 -- 172.21.15.113:0/2725559699 shutdown_connections 2024-01-15T11:25:21.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.492+0000 7f23d539b700 1 -- 172.21.15.113:0/2725559699 wait complete. 2024-01-15T11:25:21.503 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-15T11:25:21.503 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:21.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.625+0000 7fedd103c700 1 Processor -- start 2024-01-15T11:25:21.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.625+0000 7fedd103c700 1 -- start start 2024-01-15T11:25:21.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedd103c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedcc073f90 0x7fedcc074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:21.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedd103c700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedcc06e7f0 con 0x7fedcc0ce090 2024-01-15T11:25:21.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedd103c700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedcc06dec0 con 0x7fedcc15e2a0 2024-01-15T11:25:21.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedd103c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc074960 con 0x7fedcc073f90 2024-01-15T11:25:21.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedca59c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fedcc0ce090 0x7fedcc06ec80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:37470/0 (socket says 172.21.15.113:37470) 2024-01-15T11:25:21.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedca59c700 1 -- 172.21.15.113:0/3969123569 learned_addr learned my addr 172.21.15.113:0/3969123569 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:21.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedcb59e700 1 --2- 172.21.15.113:0/3969123569 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedcc073f90 0x7fedcc074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:21.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedc9d9b700 1 -- 172.21.15.113:0/3969123569 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300613931 0 0) 0x7fedcc06e7f0 con 0x7fedcc0ce090 2024-01-15T11:25:21.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedc9d9b700 1 -- 172.21.15.113:0/3969123569 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb40035b0 con 0x7fedcc0ce090 2024-01-15T11:25:21.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedc9d9b700 1 -- 172.21.15.113:0/3969123569 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071696130 0 0) 0x7fedcc06dec0 con 0x7fedcc15e2a0 2024-01-15T11:25:21.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedc9d9b700 1 -- 172.21.15.113:0/3969123569 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedcc06e7f0 con 0x7fedcc15e2a0 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedcb59e700 1 -- 172.21.15.113:0/3969123569 >> v1:172.21.15.141:6789/0 conn(0x7fedcc15e2a0 legacy=0x7fedcc06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedcb59e700 1 -- 172.21.15.113:0/3969123569 >> v1:172.21.15.113:6789/0 conn(0x7fedcc0ce090 legacy=0x7fedcc06ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedc9d9b700 1 -- 172.21.15.113:0/3969123569 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2274939750 0 0) 0x7fedb40035b0 con 0x7fedcc0ce090 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedcb59e700 1 -- 172.21.15.113:0/3969123569 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedcc06e1a0 con 0x7fedcc073f90 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.626+0000 7fedcb59e700 1 --2- 172.21.15.113:0/3969123569 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedcc073f90 0x7fedcc074420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009dc0 tx=0x7fedc400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=378391fadfbc5fcc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.627+0000 7fedc9d9b700 1 -- 172.21.15.113:0/3969123569 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc400e070 con 0x7fedcc073f90 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.627+0000 7fedc9d9b700 1 -- 172.21.15.113:0/3969123569 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc4003a20 con 0x7fedcc073f90 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.627+0000 7fedc9d9b700 1 -- 172.21.15.113:0/3969123569 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc0001670 con 0x7fedcc073f90 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.627+0000 7fedd103c700 1 -- 172.21.15.113:0/3969123569 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedcc073f90 msgr2=0x7fedcc074420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.627+0000 7fedd103c700 1 --2- 172.21.15.113:0/3969123569 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedcc073f90 0x7fedcc074420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009dc0 tx=0x7fedc400b330 comp rx=0 tx=0).stop 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.627+0000 7fedd103c700 1 -- 172.21.15.113:0/3969123569 shutdown_connections 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.627+0000 7fedd103c700 1 --2- 172.21.15.113:0/3969123569 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedcc073f90 0x7fedcc074420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.627+0000 7fedd103c700 1 -- 172.21.15.113:0/3969123569 >> 172.21.15.113:0/3969123569 conn(0x7fedcc0cd570 msgr2=0x7fedcc0cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.627+0000 7fedd103c700 1 -- 172.21.15.113:0/3969123569 shutdown_connections 2024-01-15T11:25:21.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.627+0000 7fedd103c700 1 -- 172.21.15.113:0/3969123569 wait complete. 2024-01-15T11:25:21.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.629+0000 7fedd103c700 1 Processor -- start 2024-01-15T11:25:21.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.629+0000 7fedd103c700 1 -- start start 2024-01-15T11:25:21.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.629+0000 7fedd103c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedcc073f90 0x7fedcc149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:21.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.629+0000 7fedd103c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedcc0ce090 0x7fedcc0cf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:21.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.629+0000 7fedd103c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedcc15e2a0 0x7fedcc0cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:21.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.629+0000 7fedd103c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc149b50 con 0x7fedcc15e2a0 2024-01-15T11:25:21.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.629+0000 7fedd103c700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc155c50 con 0x7fedcc0ce090 2024-01-15T11:25:21.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.629+0000 7fedd103c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc1591b0 con 0x7fedcc073f90 2024-01-15T11:25:21.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.629+0000 7fedcb59e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedcc15e2a0 0x7fedcc0cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.629+0000 7fedcb59e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedcc15e2a0 0x7fedcc0cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45530/0 (socket says 172.21.15.113:45530) 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.629+0000 7fedcb59e700 1 -- 172.21.15.113:0/2588485971 learned_addr learned my addr 172.21.15.113:0/2588485971 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedca59c700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedcc0ce090 0x7fedcc0cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedcad9d700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedcc073f90 0x7fedcc149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedcb59e700 1 -- 172.21.15.113:0/2588485971 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedcc073f90 msgr2=0x7fedcc149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedcb59e700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedcc073f90 0x7fedcc149610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedcb59e700 1 -- 172.21.15.113:0/2588485971 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedcc0ce090 msgr2=0x7fedcc0cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedcb59e700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedcc0ce090 0x7fedcc0cf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedcb59e700 1 -- 172.21.15.113:0/2588485971 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc4009a70 con 0x7fedcc15e2a0 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedca59c700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedcc0ce090 0x7fedcc0cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedcb59e700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedcc15e2a0 0x7fedcc0cf970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fedc4015040 tx=0x7fedc4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedb37fe700 1 -- 172.21.15.113:0/2588485971 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc40043c0 con 0x7fedcc15e2a0 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedb37fe700 1 -- 172.21.15.113:0/2588485971 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc4004520 con 0x7fedcc15e2a0 2024-01-15T11:25:21.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedd103c700 1 -- 172.21.15.113:0/2588485971 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedcc159490 con 0x7fedcc15e2a0 2024-01-15T11:25:21.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedb37fe700 1 -- 172.21.15.113:0/2588485971 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4004890 con 0x7fedcc15e2a0 2024-01-15T11:25:21.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.630+0000 7fedd103c700 1 -- 172.21.15.113:0/2588485971 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedcc15f580 con 0x7fedcc15e2a0 2024-01-15T11:25:21.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.631+0000 7fedd103c700 1 -- 172.21.15.113:0/2588485971 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedcc067f60 con 0x7fedcc15e2a0 2024-01-15T11:25:21.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.631+0000 7fedb37fe700 1 -- 172.21.15.113:0/2588485971 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fedc401c460 con 0x7fedcc15e2a0 2024-01-15T11:25:21.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.632+0000 7fedb37fe700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fedb806b350 0x7fedb806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:21.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.632+0000 7fedb37fe700 1 -- 172.21.15.113:0/2588485971 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedc408b1e0 con 0x7fedcc15e2a0 2024-01-15T11:25:21.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.632+0000 7fedcad9d700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fedb806b350 0x7fedb806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:21.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.632+0000 7fedcad9d700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fedb806b350 0x7fedb806d810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fedbc001220 tx=0x7fedbc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:21.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.636+0000 7fedb37fe700 1 -- 172.21.15.113:0/2588485971 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fedc40560e0 con 0x7fedcc15e2a0 2024-01-15T11:25:21.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.828+0000 7fedd103c700 1 -- 172.21.15.113:0/2588485971 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedcc072e00 con 0x7fedcc15e2a0 2024-01-15T11:25:21.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.829+0000 7fedb37fe700 1 -- 172.21.15.113:0/2588485971 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7fedc4059700 con 0x7fedcc15e2a0 2024-01-15T11:25:21.828 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:21.828 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.832+0000 7fedd103c700 1 -- 172.21.15.113:0/2588485971 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fedb806b350 msgr2=0x7fedb806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.832+0000 7fedd103c700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fedb806b350 0x7fedb806d810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fedbc001220 tx=0x7fedbc009040 comp rx=0 tx=0).stop 2024-01-15T11:25:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.832+0000 7fedd103c700 1 -- 172.21.15.113:0/2588485971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedcc15e2a0 msgr2=0x7fedcc0cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.832+0000 7fedd103c700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedcc15e2a0 0x7fedcc0cf970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fedc4015040 tx=0x7fedc4004ad0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.832+0000 7fedd103c700 1 -- 172.21.15.113:0/2588485971 shutdown_connections 2024-01-15T11:25:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.832+0000 7fedd103c700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fedb806b350 0x7fedb806d810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.832+0000 7fedd103c700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedcc15e2a0 0x7fedcc0cf970 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.832+0000 7fedd103c700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedcc073f90 0x7fedcc149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.832+0000 7fedd103c700 1 --2- 172.21.15.113:0/2588485971 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedcc0ce090 0x7fedcc0cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.832+0000 7fedd103c700 1 -- 172.21.15.113:0/2588485971 >> 172.21.15.113:0/2588485971 conn(0x7fedcc0cd570 msgr2=0x7fedcc0cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.832+0000 7fedd103c700 1 -- 172.21.15.113:0/2588485971 shutdown_connections 2024-01-15T11:25:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:21.832+0000 7fedd103c700 1 -- 172.21.15.113:0/2588485971 wait complete. 2024-01-15T11:25:21.843 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-15T11:25:21.843 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-15T11:25:21.844 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-15T11:25:21.844 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-15T11:25:21.845 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-15T11:25:21.845 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-15T11:25:21.845 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-15T11:25:21.846 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-15T11:25:22.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.043+0000 7f436f114700 1 Processor -- start 2024-01-15T11:25:22.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.043+0000 7f436f114700 1 -- start start 2024-01-15T11:25:22.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.045+0000 7f436f114700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f43680b4260 0x7f436815d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.045+0000 7f436f114700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f43680714d0 0x7f4368071950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.045+0000 7f436f114700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436815d970 con 0x7f43680d0520 2024-01-15T11:25:22.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.045+0000 7f436f114700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4368072d60 con 0x7f43680b4260 2024-01-15T11:25:22.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.045+0000 7f436f114700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436806de80 con 0x7f43680714d0 2024-01-15T11:25:22.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.045+0000 7f436ceb0700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f43680d0520 0x7f43680b3b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:35254/0 (socket says 172.21.15.113:35254) 2024-01-15T11:25:22.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.045+0000 7f436ceb0700 1 -- 172.21.15.113:0/3456863826 learned_addr learned my addr 172.21.15.113:0/3456863826 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:22.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.046+0000 7f4367fff700 1 --2- 172.21.15.113:0/3456863826 >> v2:172.21.15.113:3300/0 conn(0x7f43680b4260 0x7f436815d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.046+0000 7f436d6b1700 1 --2- 172.21.15.113:0/3456863826 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f43680714d0 0x7f4368071950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.046+0000 7f4367fff700 1 -- 172.21.15.113:0/3456863826 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f43680714d0 msgr2=0x7f4368071950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.046+0000 7f4367fff700 1 --2- 172.21.15.113:0/3456863826 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f43680714d0 0x7f4368071950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.046+0000 7f4367fff700 1 -- 172.21.15.113:0/3456863826 >> v1:172.21.15.141:6789/0 conn(0x7f43680d0520 legacy=0x7f43680b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.046+0000 7f4367fff700 1 -- 172.21.15.113:0/3456863826 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4368071e90 con 0x7f43680b4260 2024-01-15T11:25:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.046+0000 7f4367fff700 1 --2- 172.21.15.113:0/3456863826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43680b4260 0x7f436815d430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4360009dc0 tx=0x7f436000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4011f8dfbb90b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.047+0000 7f43677fe700 1 -- 172.21.15.113:0/3456863826 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4360014ce0 con 0x7f43680b4260 2024-01-15T11:25:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.047+0000 7f43677fe700 1 -- 172.21.15.113:0/3456863826 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4360014e40 con 0x7f43680b4260 2024-01-15T11:25:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.047+0000 7f43677fe700 1 -- 172.21.15.113:0/3456863826 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436000baf0 con 0x7f43680b4260 2024-01-15T11:25:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.048+0000 7f436f114700 1 -- 172.21.15.113:0/3456863826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43680b4260 msgr2=0x7f436815d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.048+0000 7f436f114700 1 --2- 172.21.15.113:0/3456863826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43680b4260 0x7f436815d430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4360009dc0 tx=0x7f436000b330 comp rx=0 tx=0).stop 2024-01-15T11:25:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.048+0000 7f436f114700 1 -- 172.21.15.113:0/3456863826 shutdown_connections 2024-01-15T11:25:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.048+0000 7f436f114700 1 --2- 172.21.15.113:0/3456863826 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f43680714d0 0x7f4368071950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.048+0000 7f436f114700 1 --2- 172.21.15.113:0/3456863826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43680b4260 0x7f436815d430 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.048+0000 7f436f114700 1 -- 172.21.15.113:0/3456863826 >> 172.21.15.113:0/3456863826 conn(0x7f43680cfa30 msgr2=0x7f43680cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.048+0000 7f436f114700 1 -- 172.21.15.113:0/3456863826 shutdown_connections 2024-01-15T11:25:22.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.049+0000 7f436f114700 1 -- 172.21.15.113:0/3456863826 wait complete. 2024-01-15T11:25:22.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.050+0000 7f436f114700 1 Processor -- start 2024-01-15T11:25:22.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.051+0000 7f436f114700 1 -- start start 2024-01-15T11:25:22.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.052+0000 7f436f114700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43680714d0 0x7f4368156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.052+0000 7f436f114700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f43680b4260 0x7f43681ea200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.052+0000 7f436f114700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43680d0520 0x7f43681589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.052+0000 7f436f114700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43680cd980 con 0x7f43680714d0 2024-01-15T11:25:22.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.052+0000 7f436f114700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43681592d0 con 0x7f43680d0520 2024-01-15T11:25:22.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.052+0000 7f436f114700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43681595b0 con 0x7f43680b4260 2024-01-15T11:25:22.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.053+0000 7f436d6b1700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43680d0520 0x7f43681589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.053+0000 7f436d6b1700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43680d0520 0x7f43681589f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51280/0 (socket says 172.21.15.113:51280) 2024-01-15T11:25:22.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.053+0000 7f436d6b1700 1 -- 172.21.15.113:0/2058816143 learned_addr learned my addr 172.21.15.113:0/2058816143 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.053+0000 7f4367fff700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f43680b4260 0x7f43681ea200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.053+0000 7f436ceb0700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43680714d0 0x7f4368156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.053+0000 7f4367fff700 1 -- 172.21.15.113:0/2058816143 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43680d0520 msgr2=0x7f43681589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.053+0000 7f4367fff700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43680d0520 0x7f43681589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.053+0000 7f4367fff700 1 -- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43680714d0 msgr2=0x7f4368156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.053+0000 7f4367fff700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43680714d0 0x7f4368156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.053+0000 7f4367fff700 1 -- 172.21.15.113:0/2058816143 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4360009a70 con 0x7f43680b4260 2024-01-15T11:25:22.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.054+0000 7f4367fff700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f43680b4260 0x7f43681ea200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f436001e000 tx=0x7f4360004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.056+0000 7fd91cac6700 1 Processor -- start 2024-01-15T11:25:22.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.056+0000 7fd91cac6700 1 -- start start 2024-01-15T11:25:22.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.057+0000 7fd91cac6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd91815ff10 0x7fd9180c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.057+0000 7fd91cac6700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd9180b3370 0x7fd9180c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.057+0000 7fd91cac6700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd918162a00 con 0x7fd9180cc330 2024-01-15T11:25:22.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.057+0000 7fd91cac6700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd918162c40 con 0x7fd9180b3370 2024-01-15T11:25:22.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.057+0000 7fd91cac6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd918162f20 con 0x7fd91815ff10 2024-01-15T11:25:22.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.057+0000 7fe37c4ac700 1 Processor -- start 2024-01-15T11:25:22.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.057+0000 7fe37c4ac700 1 -- start start 2024-01-15T11:25:22.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.057+0000 7fd916d9d700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd9180b3370 0x7fd9180c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.058+0000 7fd916d9d700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd9180b3370 0x7fd9180c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51286/0 (socket says 172.21.15.113:51286) 2024-01-15T11:25:22.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.058+0000 7fd916d9d700 1 -- 172.21.15.113:0/4232568557 learned_addr learned my addr 172.21.15.113:0/4232568557 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.058+0000 7fd916d9d700 1 -- 172.21.15.113:0/4232568557 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd91815ff10 msgr2=0x7fd9180c4cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T11:25:22.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.058+0000 7fd916d9d700 1 --2- 172.21.15.113:0/4232568557 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd91815ff10 0x7fd9180c4cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.058+0000 7fd916d9d700 1 -- 172.21.15.113:0/4232568557 >> v1:172.21.15.141:6789/0 conn(0x7fd9180cc330 legacy=0x7fd9180b2c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T11:25:22.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.058+0000 7fd916d9d700 1 -- 172.21.15.113:0/4232568557 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd918163200 con 0x7fd9180b3370 2024-01-15T11:25:22.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.058+0000 7fd916d9d700 1 --2- 172.21.15.113:0/4232568557 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9180b3370 0x7fd9180c51f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd910009e50 tx=0x7fd91000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c296ef547c1cc404 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.058+0000 7fd91559a700 1 -- 172.21.15.113:0/4232568557 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91000e070 con 0x7fd9180b3370 2024-01-15T11:25:22.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.058+0000 7fd91559a700 1 -- 172.21.15.113:0/4232568557 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910003a20 con 0x7fd9180b3370 2024-01-15T11:25:22.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.058+0000 7fd91559a700 1 -- 172.21.15.113:0/4232568557 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910003dd0 con 0x7fd9180b3370 2024-01-15T11:25:22.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.059+0000 7fd91cac6700 1 -- 172.21.15.113:0/4232568557 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9180b3370 msgr2=0x7fd9180c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.059+0000 7fd91cac6700 1 --2- 172.21.15.113:0/4232568557 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9180b3370 0x7fd9180c51f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd910009e50 tx=0x7fd91000b330 comp rx=0 tx=0).stop 2024-01-15T11:25:22.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.059+0000 7fe37c4ac700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe3740cc3c0 0x7fe3740b4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.059+0000 7fe37c4ac700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe374072850 0x7fe3740cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.059+0000 7fe37c4ac700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe37406ecd0 con 0x7fe3741456b0 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.059+0000 7fe37c4ac700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe374145a90 con 0x7fe374072850 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.059+0000 7fe37c4ac700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe37406f4e0 con 0x7fe3740cc3c0 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.059+0000 7fe37aa49700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe374072850 0x7fe3740cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.059+0000 7fe37aa49700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe374072850 0x7fe3740cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45562/0 (socket says 172.21.15.113:45562) 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.059+0000 7fe37aa49700 1 -- 172.21.15.113:0/3221760999 learned_addr learned my addr 172.21.15.113:0/3221760999 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fe37aa49700 1 -- 172.21.15.113:0/3221760999 >> v2:172.21.15.141:3300/0 conn(0x7fe3740cc3c0 msgr2=0x7fe3740b4260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fe37aa49700 1 --2- 172.21.15.113:0/3221760999 >> v2:172.21.15.141:3300/0 conn(0x7fe3740cc3c0 0x7fe3740b4260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fe37aa49700 1 -- 172.21.15.113:0/3221760999 >> v1:172.21.15.113:6789/0 conn(0x7fe3741456b0 legacy=0x7fe3740b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fe37aa49700 1 -- 172.21.15.113:0/3221760999 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3740cc8c0 con 0x7fe374072850 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fd91cac6700 1 -- 172.21.15.113:0/4232568557 shutdown_connections 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fd91cac6700 1 --2- 172.21.15.113:0/4232568557 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9180b3370 0x7fd9180c51f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fd91cac6700 1 --2- 172.21.15.113:0/4232568557 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd91815ff10 0x7fd9180c4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fd91cac6700 1 -- 172.21.15.113:0/4232568557 >> 172.21.15.113:0/4232568557 conn(0x7fd9180731a0 msgr2=0x7fd91809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fe37aa49700 1 --2- 172.21.15.113:0/3221760999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe374072850 0x7fe3740cdf30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe374062c70 tx=0x7fe364009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3283e00c1773ed server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fd91cac6700 1 -- 172.21.15.113:0/4232568557 shutdown_connections 2024-01-15T11:25:22.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fd91cac6700 1 -- 172.21.15.113:0/4232568557 wait complete. 2024-01-15T11:25:22.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fe379246700 1 -- 172.21.15.113:0/3221760999 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe364004230 con 0x7fe374072850 2024-01-15T11:25:22.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fe379246700 1 -- 172.21.15.113:0/3221760999 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe364004390 con 0x7fe374072850 2024-01-15T11:25:22.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.060+0000 7fe379246700 1 -- 172.21.15.113:0/3221760999 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3640036a0 con 0x7fe374072850 2024-01-15T11:25:22.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.061+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3221760999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe374072850 msgr2=0x7fe3740cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.061+0000 7fe37c4ac700 1 --2- 172.21.15.113:0/3221760999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe374072850 0x7fe3740cdf30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe374062c70 tx=0x7fe364009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.061+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3221760999 shutdown_connections 2024-01-15T11:25:22.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.061+0000 7fe37c4ac700 1 --2- 172.21.15.113:0/3221760999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe374072850 0x7fe3740cdf30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.061+0000 7fe37c4ac700 1 --2- 172.21.15.113:0/3221760999 >> v2:172.21.15.141:3300/0 conn(0x7fe3740cc3c0 0x7fe3740b4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.061+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3221760999 >> 172.21.15.113:0/3221760999 conn(0x7fe3740cfa30 msgr2=0x7fe3740cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.062+0000 7fd91cac6700 1 Processor -- start 2024-01-15T11:25:22.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.062+0000 7fd91cac6700 1 -- start start 2024-01-15T11:25:22.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.063+0000 7f43657fa700 1 -- 172.21.15.113:0/2058816143 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4360003e50 con 0x7f43680b4260 2024-01-15T11:25:22.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.063+0000 7f43657fa700 1 -- 172.21.15.113:0/2058816143 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4360009410 con 0x7f43680b4260 2024-01-15T11:25:22.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.063+0000 7f43657fa700 1 -- 172.21.15.113:0/2058816143 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43600145e0 con 0x7f43680b4260 2024-01-15T11:25:22.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.063+0000 7fd91cac6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9180b3370 0x7fd91815f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.063+0000 7fd91cac6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd91814b580 0x7fd91814de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.063+0000 7fd91cac6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd91814ba00 0x7fd9181617f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.063+0000 7fd91cac6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9180c67b0 con 0x7fd9180b3370 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.063+0000 7fd91cac6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91815f5a0 con 0x7fd91814ba00 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.063+0000 7fd91cac6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9180c7c10 con 0x7fd91814b580 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.064+0000 7fd91659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9180b3370 0x7fd91815f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.064+0000 7fd91659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9180b3370 0x7fd91815f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45570/0 (socket says 172.21.15.113:45570) 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.064+0000 7fd91659c700 1 -- 172.21.15.113:0/2581555752 learned_addr learned my addr 172.21.15.113:0/2581555752 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.064+0000 7fd915d9b700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd91814b580 0x7fd91814de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.064+0000 7fd916d9d700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd91814ba00 0x7fd9181617f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.064+0000 7fd916d9d700 1 -- 172.21.15.113:0/2581555752 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd91814b580 msgr2=0x7fd91814de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.064+0000 7fd916d9d700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd91814b580 0x7fd91814de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.064+0000 7fd916d9d700 1 -- 172.21.15.113:0/2581555752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9180b3370 msgr2=0x7fd91815f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.064+0000 7fd916d9d700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9180b3370 0x7fd91815f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.064+0000 7fd916d9d700 1 -- 172.21.15.113:0/2581555752 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd910009b30 con 0x7fd91814ba00 2024-01-15T11:25:22.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.063+0000 7f436f114700 1 -- 172.21.15.113:0/2058816143 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43680cd610 con 0x7f43680b4260 2024-01-15T11:25:22.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.063+0000 7f436f114700 1 -- 172.21.15.113:0/2058816143 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4368158f30 con 0x7f43680b4260 2024-01-15T11:25:22.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.064+0000 7ff52eff6700 1 Processor -- start 2024-01-15T11:25:22.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.064+0000 7ff52eff6700 1 -- start start 2024-01-15T11:25:22.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.065+0000 7fd916d9d700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd91814ba00 0x7fd9181617f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd910014000 tx=0x7fd910004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.065+0000 7fa3ec216700 1 Processor -- start 2024-01-15T11:25:22.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.065+0000 7f436f114700 1 -- 172.21.15.113:0/2058816143 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4348000fc0 con 0x7f43680b4260 2024-01-15T11:25:22.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7ff52eff6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5280b7430 0x7ff5280d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7ff52eff6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff5280c7560 0x7ff5280d1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7ff52eff6700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5280ceaa0 con 0x7ff5280b33d0 2024-01-15T11:25:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7ff52eff6700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5280ced80 con 0x7ff5280c7560 2024-01-15T11:25:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7ff52eff6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5280cf060 con 0x7ff5280b7430 2024-01-15T11:25:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7ff52d593700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff5280c7560 0x7ff5280d1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7ff52d593700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff5280c7560 0x7ff5280d1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45574/0 (socket says 172.21.15.113:45574) 2024-01-15T11:25:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7ff52d593700 1 -- 172.21.15.113:0/365410611 learned_addr learned my addr 172.21.15.113:0/365410611 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7ff51ffff700 1 --2- 172.21.15.113:0/365410611 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5280b7430 0x7ff5280d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.065+0000 7f54c9f10700 1 Processor -- start 2024-01-15T11:25:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.065+0000 7f54c9f10700 1 -- start start 2024-01-15T11:25:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7f54c9f10700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54c40cf440 0x7f54c406e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7f54c9f10700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c4072110 con 0x7f54c41665a0 2024-01-15T11:25:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7f54c9f10700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c406ee00 con 0x7f54c4070fd0 2024-01-15T11:25:22.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7f54c9f10700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c4071500 con 0x7f54c40cf440 2024-01-15T11:25:22.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.065+0000 7fd906ffd700 1 -- 172.21.15.113:0/2581555752 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9100092e0 con 0x7fd91814ba00 2024-01-15T11:25:22.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.065+0000 7fd906ffd700 1 -- 172.21.15.113:0/2581555752 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910004240 con 0x7fd91814ba00 2024-01-15T11:25:22.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.065+0000 7fd906ffd700 1 -- 172.21.15.113:0/2581555752 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9100045f0 con 0x7fd91814ba00 2024-01-15T11:25:22.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.065+0000 7fd91cac6700 1 -- 172.21.15.113:0/2581555752 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9180c7ef0 con 0x7fd91814ba00 2024-01-15T11:25:22.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.065+0000 7fd91cac6700 1 -- 172.21.15.113:0/2581555752 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9180c84b0 con 0x7fd91814ba00 2024-01-15T11:25:22.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7f54c3fff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f54c41665a0 0x7f54c40b2a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:37500/0 (socket says 172.21.15.113:37500) 2024-01-15T11:25:22.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7f54c3fff700 1 -- 172.21.15.113:0/22787067 learned_addr learned my addr 172.21.15.113:0/22787067 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:22.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7ff51ffff700 1 -- 172.21.15.113:0/365410611 >> v2:172.21.15.113:3300/0 conn(0x7ff5280c7560 msgr2=0x7ff5280d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7ff51ffff700 1 --2- 172.21.15.113:0/365410611 >> v2:172.21.15.113:3300/0 conn(0x7ff5280c7560 0x7ff5280d1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7ff51ffff700 1 -- 172.21.15.113:0/365410611 >> v1:172.21.15.141:6789/0 conn(0x7ff5280b33d0 legacy=0x7ff5280b6d20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T11:25:22.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7ff51ffff700 1 -- 172.21.15.113:0/365410611 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5280cf340 con 0x7ff5280b7430 2024-01-15T11:25:22.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7f54c37fe700 1 --2- 172.21.15.113:0/22787067 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54c40cf440 0x7f54c406e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.066+0000 7fa3ec216700 1 -- start start 2024-01-15T11:25:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7fa3ec216700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa3e40d3080 0x7fa3e4163580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7fa3ec216700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa3e40b6d20 0x7fa3e40b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7fa3ec216700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e40d3460 con 0x7fa3e40b9bf0 2024-01-15T11:25:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7fa3ec216700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e40b76f0 con 0x7fa3e40d3080 2024-01-15T11:25:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7fa3ec216700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e40b3170 con 0x7fa3e40b6d20 2024-01-15T11:25:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7f54c27fc700 1 -- 172.21.15.113:0/22787067 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106890611 0 0) 0x7f54c406ee00 con 0x7f54c4070fd0 2024-01-15T11:25:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7f54c27fc700 1 -- 172.21.15.113:0/22787067 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ac0035b0 con 0x7f54c4070fd0 2024-01-15T11:25:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7f54c27fc700 1 -- 172.21.15.113:0/22787067 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822771911 0 0) 0x7f54c4072110 con 0x7f54c41665a0 2024-01-15T11:25:22.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7f54c27fc700 1 -- 172.21.15.113:0/22787067 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c406ee00 con 0x7f54c41665a0 2024-01-15T11:25:22.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7ff52d593700 1 --2- 172.21.15.113:0/365410611 >> v2:172.21.15.113:3300/0 conn(0x7ff5280c7560 0x7ff5280d1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:22.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7ff51ffff700 1 --2- 172.21.15.113:0/365410611 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5280b7430 0x7ff5280d1aa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff518009840 tx=0x7ff518009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8241f9fbea00465a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7f54c37fe700 1 -- 172.21.15.113:0/22787067 >> v1:172.21.15.141:6789/0 conn(0x7f54c4070fd0 legacy=0x7f54c4165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7f54c37fe700 1 -- 172.21.15.113:0/22787067 >> v1:172.21.15.113:6789/0 conn(0x7f54c41665a0 legacy=0x7f54c40b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7f54c37fe700 1 -- 172.21.15.113:0/22787067 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c40717e0 con 0x7f54c40cf440 2024-01-15T11:25:22.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7fd906ffd700 1 -- 172.21.15.113:0/2581555752 <== mon.1 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd910019070 con 0x7fd91814ba00 2024-01-15T11:25:22.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7f54c37fe700 1 --2- 172.21.15.113:0/22787067 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54c40cf440 0x7f54c406e8c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f54b4009870 tx=0x7f54b4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ce7e03542c2f1bd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7ff51f7fe700 1 -- 172.21.15.113:0/365410611 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518004230 con 0x7ff5280b7430 2024-01-15T11:25:22.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7ff51f7fe700 1 -- 172.21.15.113:0/365410611 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff518004390 con 0x7ff5280b7430 2024-01-15T11:25:22.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7ff51f7fe700 1 -- 172.21.15.113:0/365410611 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5180036a0 con 0x7ff5280b7430 2024-01-15T11:25:22.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7fd906ffd700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd8f806b3a0 0x7fd8f806d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.067+0000 7f43657fa700 1 -- 172.21.15.113:0/2058816143 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4360013030 con 0x7f43680b4260 2024-01-15T11:25:22.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7f43657fa700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f435406b420 0x7f435406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7f43657fa700 1 -- 172.21.15.113:0/2058816143 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f436008b5c0 con 0x7f43680b4260 2024-01-15T11:25:22.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7f43657fa700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:6816/1186622624,v1:172.21.15.113:6817/1186622624] conn(0x7f4354073640 0x7f4354075a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7f43657fa700 1 -- 172.21.15.113:0/2058816143 --> [v2:172.21.15.113:6816/1186622624,v1:172.21.15.113:6817/1186622624] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4354076110 con 0x7f4354073640 2024-01-15T11:25:22.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7f43657fa700 1 -- 172.21.15.113:0/2058816143 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f436008b970 con 0x7f43680b4260 2024-01-15T11:25:22.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7f436ceb0700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f435406b420 0x7f435406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7ff52eff6700 1 -- 172.21.15.113:0/365410611 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5280b7430 msgr2=0x7ff5280d1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7ff52eff6700 1 --2- 172.21.15.113:0/365410611 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5280b7430 0x7ff5280d1aa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff518009840 tx=0x7ff518009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:22.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7fd906ffd700 1 -- 172.21.15.113:0/2581555752 <== mon.1 v2:172.21.15.141:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd91008a690 con 0x7fd91814ba00 2024-01-15T11:25:22.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7fd91659c700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd8f806b3a0 0x7fd8f806d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3221760999 shutdown_connections 2024-01-15T11:25:22.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7f436d6b1700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:6816/1186622624,v1:172.21.15.113:6817/1186622624] conn(0x7f4354073640 0x7f4354075a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7ff52eff6700 1 -- 172.21.15.113:0/365410611 shutdown_connections 2024-01-15T11:25:22.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7ff52eff6700 1 --2- 172.21.15.113:0/365410611 >> v2:172.21.15.113:3300/0 conn(0x7ff5280c7560 0x7ff5280d1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7ff52eff6700 1 --2- 172.21.15.113:0/365410611 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5280b7430 0x7ff5280d1aa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7ff52eff6700 1 -- 172.21.15.113:0/365410611 >> 172.21.15.113:0/365410611 conn(0x7ff5280d3ed0 msgr2=0x7ff5280d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7f54c27fc700 1 -- 172.21.15.113:0/22787067 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b4004230 con 0x7f54c40cf440 2024-01-15T11:25:22.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7f54c27fc700 1 -- 172.21.15.113:0/22787067 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b4004390 con 0x7f54c40cf440 2024-01-15T11:25:22.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.068+0000 7f54c27fc700 1 -- 172.21.15.113:0/22787067 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b40036a0 con 0x7f54c40cf440 2024-01-15T11:25:22.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7f54c9f10700 1 -- 172.21.15.113:0/22787067 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54c40cf440 msgr2=0x7f54c406e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7f54c9f10700 1 --2- 172.21.15.113:0/22787067 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54c40cf440 0x7f54c406e8c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f54b4009870 tx=0x7f54b4009b80 comp rx=0 tx=0).stop 2024-01-15T11:25:22.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7f54c9f10700 1 -- 172.21.15.113:0/22787067 shutdown_connections 2024-01-15T11:25:22.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7f54c9f10700 1 --2- 172.21.15.113:0/22787067 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54c40cf440 0x7f54c406e8c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7f54c9f10700 1 -- 172.21.15.113:0/22787067 >> 172.21.15.113:0/22787067 conn(0x7f54c40b1740 msgr2=0x7f54c40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7fd91659c700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd8f806b3a0 0x7fd8f806d860 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009a10 tx=0x7fd90c0091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7f436ceb0700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f435406b420 0x7f435406d8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4358005910 tx=0x7f4358009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7f436d6b1700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:6816/1186622624,v1:172.21.15.113:6817/1186622624] conn(0x7f4354073640 0x7f4354075a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7fa3ea7b3700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa3e40b6d20 0x7fa3e40b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7fa3ea7b3700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa3e40b6d20 0x7fa3e40b71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:47310/0 (socket says 172.21.15.113:47310) 2024-01-15T11:25:22.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7fa3ea7b3700 1 -- 172.21.15.113:0/1840794115 learned_addr learned my addr 172.21.15.113:0/1840794115 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7fa3e9fb2700 1 --2- 172.21.15.113:0/1840794115 >> v2:172.21.15.141:3300/0 conn(0x7fa3e40d3080 0x7fa3e4163580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7f54c9f10700 1 -- 172.21.15.113:0/22787067 shutdown_connections 2024-01-15T11:25:22.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7fd91cac6700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.141:6808/1489198687,v1:172.21.15.141:6809/1489198687] conn(0x7fd8fc0015e0 0x7fd8fc003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7fd91cac6700 1 -- 172.21.15.113:0/2581555752 --> [v2:172.21.15.141:6808/1489198687,v1:172.21.15.141:6809/1489198687] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8fc006bd0 con 0x7fd8fc0015e0 2024-01-15T11:25:22.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7fd915d9b700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.141:6808/1489198687,v1:172.21.15.141:6809/1489198687] conn(0x7fd8fc0015e0 0x7fd8fc003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7f54c9f10700 1 -- 172.21.15.113:0/22787067 wait complete. 2024-01-15T11:25:22.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.071+0000 7ff52eff6700 1 -- 172.21.15.113:0/365410611 shutdown_connections 2024-01-15T11:25:22.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.071+0000 7f54c9f10700 1 Processor -- start 2024-01-15T11:25:22.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.071+0000 7fd915d9b700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.141:6808/1489198687,v1:172.21.15.141:6809/1489198687] conn(0x7fd8fc0015e0 0x7fd8fc003aa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.072+0000 7fd906ffd700 1 -- 172.21.15.113:0/2581555752 <== osd.5 v2:172.21.15.141:6808/1489198687 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd8fc006bd0 con 0x7fd8fc0015e0 2024-01-15T11:25:22.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.071+0000 7ff52eff6700 1 -- 172.21.15.113:0/365410611 wait complete. 2024-01-15T11:25:22.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.072+0000 7ff52eff6700 1 Processor -- start 2024-01-15T11:25:22.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.071+0000 7f54c9f10700 1 -- start start 2024-01-15T11:25:22.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.072+0000 7f54c9f10700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54c4070fd0 0x7f54c4204740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.072+0000 7f54c9f10700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54c40cf440 0x7f54c415ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.072+0000 7f54c9f10700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54c41665a0 0x7f54c4200080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.072+0000 7f54c9f10700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c4204d20 con 0x7f54c40cf440 2024-01-15T11:25:22.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.072+0000 7f54c9f10700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c42005c0 con 0x7f54c41665a0 2024-01-15T11:25:22.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.072+0000 7f54c9f10700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c4156ed0 con 0x7f54c4070fd0 2024-01-15T11:25:22.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7f43657fa700 1 -- 172.21.15.113:0/2058816143 <== osd.2 v2:172.21.15.113:6816/1186622624 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4354076110 con 0x7f4354073640 2024-01-15T11:25:22.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7fa3ea7b3700 1 -- 172.21.15.113:0/1840794115 >> v2:172.21.15.141:3300/0 conn(0x7fa3e40d3080 msgr2=0x7fa3e4163580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7fa3ea7b3700 1 --2- 172.21.15.113:0/1840794115 >> v2:172.21.15.141:3300/0 conn(0x7fa3e40d3080 0x7fa3e4163580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7fa3ea7b3700 1 -- 172.21.15.113:0/1840794115 >> v1:172.21.15.141:6789/0 conn(0x7fa3e40b9bf0 legacy=0x7fa3e40b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7fa3ea7b3700 1 -- 172.21.15.113:0/1840794115 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e40b3390 con 0x7fa3e40b6d20 2024-01-15T11:25:22.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7fa3ea7b3700 1 --2- 172.21.15.113:0/1840794115 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa3e40b6d20 0x7fa3e40b71b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009e80 tx=0x7fa3dc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f62a74a2e1f8eec2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7fa3e8fb0700 1 -- 172.21.15.113:0/1840794115 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc00e070 con 0x7fa3e40b6d20 2024-01-15T11:25:22.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7fa3e8fb0700 1 -- 172.21.15.113:0/1840794115 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3dc00b8d0 con 0x7fa3e40b6d20 2024-01-15T11:25:22.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.070+0000 7fa3e8fb0700 1 -- 172.21.15.113:0/1840794115 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc00bc80 con 0x7fa3e40b6d20 2024-01-15T11:25:22.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.072+0000 7fa3ec216700 1 -- 172.21.15.113:0/1840794115 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa3e40b6d20 msgr2=0x7fa3e40b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.072+0000 7fa3ec216700 1 --2- 172.21.15.113:0/1840794115 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa3e40b6d20 0x7fa3e40b71b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009e80 tx=0x7fa3dc00b330 comp rx=0 tx=0).stop 2024-01-15T11:25:22.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.069+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3221760999 wait complete. 2024-01-15T11:25:22.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.071+0000 7fe37c4ac700 1 Processor -- start 2024-01-15T11:25:22.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7f54c2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54c40cf440 0x7f54c415ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7f54c2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54c40cf440 0x7f54c415ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45578/0 (socket says 172.21.15.113:45578) 2024-01-15T11:25:22.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7f54c2ffd700 1 -- 172.21.15.113:0/2844102001 learned_addr learned my addr 172.21.15.113:0/2844102001 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7f54c3fff700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54c41665a0 0x7f54c4200080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7f54c37fe700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54c4070fd0 0x7f54c4204740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7fa3ec216700 1 -- 172.21.15.113:0/1840794115 shutdown_connections 2024-01-15T11:25:22.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7fa3ec216700 1 --2- 172.21.15.113:0/1840794115 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa3e40b6d20 0x7fa3e40b71b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7fa3ec216700 1 --2- 172.21.15.113:0/1840794115 >> v2:172.21.15.141:3300/0 conn(0x7fa3e40d3080 0x7fa3e4163580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7fa3ec216700 1 -- 172.21.15.113:0/1840794115 >> 172.21.15.113:0/1840794115 conn(0x7fa3e40d3ed0 msgr2=0x7fa3e40d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7f54c3fff700 1 -- 172.21.15.113:0/2844102001 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54c4070fd0 msgr2=0x7f54c4204740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7f54c3fff700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54c4070fd0 0x7f54c4204740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7f54c3fff700 1 -- 172.21.15.113:0/2844102001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54c40cf440 msgr2=0x7f54c415ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7f54c3fff700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54c40cf440 0x7f54c415ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7f54c3fff700 1 -- 172.21.15.113:0/2844102001 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b40094a0 con 0x7f54c41665a0 2024-01-15T11:25:22.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7f54c3fff700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54c41665a0 0x7f54c4200080 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f54c40d0ae0 tx=0x7f54bc00cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7f54abfff700 1 -- 172.21.15.113:0/2844102001 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc00a9e0 con 0x7f54c41665a0 2024-01-15T11:25:22.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7ff52eff6700 1 -- start start 2024-01-15T11:25:22.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7ff52eff6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5280b7430 0x7ff528169930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7ff52eff6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5280c7560 0x7ff5281f43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7ff52eff6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5281f6ee0 0x7ff5280719a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7ff52eff6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52816a280 con 0x7ff5280c7560 2024-01-15T11:25:22.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7ff52eff6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5281f4ac0 con 0x7ff5281f6ee0 2024-01-15T11:25:22.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7ff52eff6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5281f4da0 con 0x7ff5280b7430 2024-01-15T11:25:22.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7ff52cd92700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5280b7430 0x7ff528169930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7ff52cd92700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5280b7430 0x7ff528169930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:47326/0 (socket says 172.21.15.113:47326) 2024-01-15T11:25:22.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7ff52cd92700 1 -- 172.21.15.113:0/1456054772 learned_addr learned my addr 172.21.15.113:0/1456054772 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7ff52d593700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5281f6ee0 0x7ff5280719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.073+0000 7fe37c4ac700 1 -- start start 2024-01-15T11:25:22.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7fe37c4ac700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe374072850 0x7fe37406e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7fe37c4ac700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe3741456b0 0x7fe3741f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7fe37c4ac700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe3741556a0 0x7fe37413f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7fe37c4ac700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37413e200 con 0x7fe374072850 2024-01-15T11:25:22.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7fe37c4ac700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37413e4e0 con 0x7fe3741456b0 2024-01-15T11:25:22.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7fe37c4ac700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe37413e7c0 con 0x7fe3741556a0 2024-01-15T11:25:22.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7fe37aa49700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe3741556a0 0x7fe37413f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7fe37aa49700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe3741556a0 0x7fe37413f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:47332/0 (socket says 172.21.15.113:47332) 2024-01-15T11:25:22.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7fe37aa49700 1 -- 172.21.15.113:0/3569903455 learned_addr learned my addr 172.21.15.113:0/3569903455 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7fe379a47700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe3741456b0 0x7fe3741f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7ff52d593700 1 -- 172.21.15.113:0/1456054772 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5280b7430 msgr2=0x7ff528169930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7ff52d593700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff5280b7430 0x7ff528169930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7ff52d593700 1 -- 172.21.15.113:0/1456054772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5280c7560 msgr2=0x7ff5281f43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7ff52d593700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5280c7560 0x7ff5281f43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7ff52d593700 1 -- 172.21.15.113:0/1456054772 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff520009b30 con 0x7ff5281f6ee0 2024-01-15T11:25:22.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7f54abfff700 1 -- 172.21.15.113:0/2844102001 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54bc00ab40 con 0x7f54c41665a0 2024-01-15T11:25:22.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.074+0000 7f54abfff700 1 -- 172.21.15.113:0/2844102001 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc0070d0 con 0x7f54c41665a0 2024-01-15T11:25:22.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7ff52d593700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5281f6ee0 0x7ff5280719a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff52000ef90 tx=0x7ff52000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7fe37aa49700 1 -- 172.21.15.113:0/3569903455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe3741456b0 msgr2=0x7fe3741f79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7fe37aa49700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe3741456b0 0x7fe3741f79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7fe37aa49700 1 -- 172.21.15.113:0/3569903455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe374072850 msgr2=0x7fe37406e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7fe37aa49700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe374072850 0x7fe37406e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7fe37aa49700 1 -- 172.21.15.113:0/3569903455 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe364009550 con 0x7fe3741556a0 2024-01-15T11:25:22.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7fa3ec216700 1 -- 172.21.15.113:0/1840794115 shutdown_connections 2024-01-15T11:25:22.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7ff51d7fa700 1 -- 172.21.15.113:0/1456054772 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff520018070 con 0x7ff5281f6ee0 2024-01-15T11:25:22.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7ff51d7fa700 1 -- 172.21.15.113:0/1456054772 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52000f040 con 0x7ff5281f6ee0 2024-01-15T11:25:22.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7ff51d7fa700 1 -- 172.21.15.113:0/1456054772 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff520003c40 con 0x7ff5281f6ee0 2024-01-15T11:25:22.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.076+0000 7ff52eff6700 1 -- 172.21.15.113:0/1456054772 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5180094a0 con 0x7ff5281f6ee0 2024-01-15T11:25:22.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.076+0000 7ff52eff6700 1 -- 172.21.15.113:0/1456054772 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5281f5360 con 0x7ff5281f6ee0 2024-01-15T11:25:22.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.077+0000 7ff52eff6700 1 -- 172.21.15.113:0/1456054772 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff5280ce730 con 0x7ff5281f6ee0 2024-01-15T11:25:22.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.075+0000 7fa3ec216700 1 -- 172.21.15.113:0/1840794115 wait complete. 2024-01-15T11:25:22.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.077+0000 7fa3ec216700 1 Processor -- start 2024-01-15T11:25:22.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.077+0000 7fa3ec216700 1 -- start start 2024-01-15T11:25:22.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.078+0000 7fa3ec216700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3e40b6d20 0x7fa3e4154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.078+0000 7fa3ec216700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa3e40d3080 0x7fa3e4152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.078+0000 7fa3ec216700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3e406c210 0x7fa3e4154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.078+0000 7fa3ec216700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e4152a60 con 0x7fa3e40b6d20 2024-01-15T11:25:22.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.078+0000 7fa3ec216700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e406cb80 con 0x7fa3e406c210 2024-01-15T11:25:22.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.078+0000 7fa3ec216700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e4152410 con 0x7fa3e40d3080 2024-01-15T11:25:22.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7ff51d7fa700 1 -- 172.21.15.113:0/1456054772 <== mon.1 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff52000c930 con 0x7ff5281f6ee0 2024-01-15T11:25:22.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7ff51d7fa700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff51406b420 0x7ff51406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7fa3e97b1700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa3e40d3080 0x7fa3e4152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7fa3e97b1700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa3e40d3080 0x7fa3e4152ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:47346/0 (socket says 172.21.15.113:47346) 2024-01-15T11:25:22.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7fa3e97b1700 1 -- 172.21.15.113:0/150450123 learned_addr learned my addr 172.21.15.113:0/150450123 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7fa3ea7b3700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3e406c210 0x7fa3e4154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7fa3e9fb2700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3e40b6d20 0x7fa3e4154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7ff51d7fa700 1 -- 172.21.15.113:0/1456054772 <== mon.1 v2:172.21.15.141:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff52008a920 con 0x7ff5281f6ee0 2024-01-15T11:25:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7ff51d7fa700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.113:6808/2024634597,v1:172.21.15.113:6809/2024634597] conn(0x7ff514073640 0x7ff514075a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7ff51d7fa700 1 -- 172.21.15.113:0/1456054772 --> [v2:172.21.15.113:6808/2024634597,v1:172.21.15.113:6809/2024634597] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff514076110 con 0x7ff514073640 2024-01-15T11:25:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7ff51d7fa700 1 -- 172.21.15.113:0/1456054772 <== mon.1 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff52008acd0 con 0x7ff5281f6ee0 2024-01-15T11:25:22.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7fa3ea7b3700 1 -- 172.21.15.113:0/150450123 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa3e40d3080 msgr2=0x7fa3e4152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7fa3ea7b3700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa3e40d3080 0x7fa3e4152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7fa3ea7b3700 1 -- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3e40b6d20 msgr2=0x7fa3e4154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7fa3ea7b3700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3e40b6d20 0x7fa3e4154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7ff52cd92700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff51406b420 0x7ff51406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.079+0000 7fa3ea7b3700 1 -- 172.21.15.113:0/150450123 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc009b30 con 0x7fa3e406c210 2024-01-15T11:25:22.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.080+0000 7ff52cd92700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff51406b420 0x7ff51406d8e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff524009a30 tx=0x7ff524009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.080+0000 7f54c9f10700 1 -- 172.21.15.113:0/2844102001 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c4162fd0 con 0x7f54c41665a0 2024-01-15T11:25:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.080+0000 7f54c9f10700 1 -- 172.21.15.113:0/2844102001 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c4157450 con 0x7f54c41665a0 2024-01-15T11:25:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.080+0000 7fa3ea7b3700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3e406c210 0x7fa3e4154ae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc015000 tx=0x7fa3dc004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.080+0000 7f54c9f10700 1 -- 172.21.15.113:0/2844102001 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f54c4071da0 con 0x7f54c41665a0 2024-01-15T11:25:22.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.080+0000 7fa3da7fc700 1 -- 172.21.15.113:0/150450123 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc004800 con 0x7fa3e406c210 2024-01-15T11:25:22.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.080+0000 7fa3da7fc700 1 -- 172.21.15.113:0/150450123 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3dc0092e0 con 0x7fa3e406c210 2024-01-15T11:25:22.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.080+0000 7fa3ec216700 1 -- 172.21.15.113:0/150450123 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e41526f0 con 0x7fa3e406c210 2024-01-15T11:25:22.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.080+0000 7fa3da7fc700 1 -- 172.21.15.113:0/150450123 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc01c660 con 0x7fa3e406c210 2024-01-15T11:25:22.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.080+0000 7fa3ec216700 1 -- 172.21.15.113:0/150450123 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e4153430 con 0x7fa3e406c210 2024-01-15T11:25:22.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.081+0000 7fa3ec216700 1 -- 172.21.15.113:0/150450123 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa3e4163060 con 0x7fa3e406c210 2024-01-15T11:25:22.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.083+0000 7f7f2edb4700 1 Processor -- start 2024-01-15T11:25:22.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.082+0000 7f54abfff700 1 -- 172.21.15.113:0/2844102001 <== mon.1 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f54bc007230 con 0x7f54c41665a0 2024-01-15T11:25:22.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.082+0000 7f54abfff700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f54b006b350 0x7f54b006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.082+0000 7f54abfff700 1 -- 172.21.15.113:0/2844102001 <== mon.1 v2:172.21.15.141:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54bc089b60 con 0x7f54c41665a0 2024-01-15T11:25:22.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.083+0000 7f54abfff700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.141:6800/1209001813,v1:172.21.15.141:6801/1209001813] conn(0x7f54b00734f0 0x7f54b0075910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.083+0000 7f54abfff700 1 -- 172.21.15.113:0/2844102001 --> [v2:172.21.15.141:6800/1209001813,v1:172.21.15.141:6801/1209001813] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54b0075fc0 con 0x7f54b00734f0 2024-01-15T11:25:22.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.083+0000 7f54abfff700 1 -- 172.21.15.113:0/2844102001 <== mon.1 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54bc089ed0 con 0x7f54c41665a0 2024-01-15T11:25:22.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.083+0000 7f54c37fe700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f54b006b350 0x7f54b006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.083+0000 7fe37aa49700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe3741556a0 0x7fe37413f580 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe3740737d0 tx=0x7fe364004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.083+0000 7f7f2edb4700 1 -- start start 2024-01-15T11:25:22.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.083+0000 7f54c2ffd700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.141:6800/1209001813,v1:172.21.15.141:6801/1209001813] conn(0x7f54b00734f0 0x7f54b0075910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.083+0000 7f7f2edb4700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7f28162240 0x7f7f280afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.083+0000 7f7f2edb4700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7f280ae4e0 0x7f7f2806c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f7f2edb4700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f280b46e0 con 0x7f7f2806f360 2024-01-15T11:25:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f7f2edb4700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f28166f50 con 0x7f7f28162240 2024-01-15T11:25:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f7f2edb4700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f280b49c0 con 0x7f7f280ae4e0 2024-01-15T11:25:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f7f27fff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7f280ae4e0 0x7f7f2806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7ff51ffff700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.113:6808/2024634597,v1:172.21.15.113:6809/2024634597] conn(0x7ff514073640 0x7ff514075a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f7f27fff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7f280ae4e0 0x7f7f2806c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51360/0 (socket says 172.21.15.113:51360) 2024-01-15T11:25:22.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f7f27fff700 1 -- 172.21.15.113:0/2688521614 learned_addr learned my addr 172.21.15.113:0/2688521614 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f7f27fff700 1 -- 172.21.15.113:0/2688521614 >> v2:172.21.15.113:3300/0 conn(0x7f7f28162240 msgr2=0x7f7f280afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f7f27fff700 1 --2- 172.21.15.113:0/2688521614 >> v2:172.21.15.113:3300/0 conn(0x7f7f28162240 0x7f7f280afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f7f27fff700 1 -- 172.21.15.113:0/2688521614 >> v1:172.21.15.141:6789/0 conn(0x7f7f2806f360 legacy=0x7f7f28073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f7f27fff700 1 -- 172.21.15.113:0/2688521614 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f280b4ca0 con 0x7f7f280ae4e0 2024-01-15T11:25:22.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7fe36affd700 1 -- 172.21.15.113:0/3569903455 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe364011630 con 0x7fe3741556a0 2024-01-15T11:25:22.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7fe36affd700 1 -- 172.21.15.113:0/3569903455 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe364011790 con 0x7fe3741556a0 2024-01-15T11:25:22.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7fe36affd700 1 -- 172.21.15.113:0/3569903455 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe364011b40 con 0x7fe3741556a0 2024-01-15T11:25:22.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f54c2ffd700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.141:6800/1209001813,v1:172.21.15.141:6801/1209001813] conn(0x7f54b00734f0 0x7f54b0075910 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f54c37fe700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f54b006b350 0x7f54b006d810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f54b40097c0 tx=0x7f54b401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7f7f27fff700 1 --2- 172.21.15.113:0/2688521614 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f280ae4e0 0x7f7f2806c000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7f280b57b0 tx=0x7f7f14009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3938e0ada56aba4b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7ff51ffff700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.113:6808/2024634597,v1:172.21.15.113:6809/2024634597] conn(0x7ff514073640 0x7ff514075a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.085+0000 7fa3da7fc700 1 -- 172.21.15.113:0/150450123 <== mon.1 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa3dc01c7c0 con 0x7fa3e406c210 2024-01-15T11:25:22.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3569903455 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe37413eaa0 con 0x7fe3741556a0 2024-01-15T11:25:22.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.084+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3569903455 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe37413f060 con 0x7fe3741556a0 2024-01-15T11:25:22.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.086+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3569903455 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe35c000fc0 con 0x7fe3741556a0 2024-01-15T11:25:22.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.086+0000 7f7f277fe700 1 -- 172.21.15.113:0/2688521614 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1400b870 con 0x7f7f280ae4e0 2024-01-15T11:25:22.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.086+0000 7f7f277fe700 1 -- 172.21.15.113:0/2688521614 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1400b9d0 con 0x7f7f280ae4e0 2024-01-15T11:25:22.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.086+0000 7f7f277fe700 1 -- 172.21.15.113:0/2688521614 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1400bd60 con 0x7f7f280ae4e0 2024-01-15T11:25:22.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.086+0000 7fa3da7fc700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa3c806b350 0x7fa3c806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.086+0000 7fa3da7fc700 1 -- 172.21.15.113:0/150450123 <== mon.1 v2:172.21.15.141:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3dc08b930 con 0x7fa3e406c210 2024-01-15T11:25:22.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.086+0000 7fa3da7fc700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:6824/2814390340,v1:172.21.15.113:6825/2814390340] conn(0x7fa3c80734f0 0x7fa3c8075910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.086+0000 7fa3da7fc700 1 -- 172.21.15.113:0/150450123 --> [v2:172.21.15.113:6824/2814390340,v1:172.21.15.113:6825/2814390340] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3c8075fc0 con 0x7fa3c80734f0 2024-01-15T11:25:22.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.086+0000 7fa3da7fc700 1 -- 172.21.15.113:0/150450123 <== mon.1 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3dc08bd80 con 0x7fa3e406c210 2024-01-15T11:25:22.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.086+0000 7fa3e9fb2700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa3c806b350 0x7fa3c806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.086+0000 7fa3e97b1700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:6824/2814390340,v1:172.21.15.113:6825/2814390340] conn(0x7fa3c80734f0 0x7fa3c8075910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.087+0000 7f7f2edb4700 1 -- 172.21.15.113:0/2688521614 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f280ae4e0 msgr2=0x7f7f2806c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.087+0000 7f7f2edb4700 1 --2- 172.21.15.113:0/2688521614 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f280ae4e0 0x7f7f2806c000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7f280b57b0 tx=0x7f7f14009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.087+0000 7f7f2edb4700 1 -- 172.21.15.113:0/2688521614 shutdown_connections 2024-01-15T11:25:22.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.087+0000 7f7f2edb4700 1 --2- 172.21.15.113:0/2688521614 >> v2:172.21.15.113:3300/0 conn(0x7f7f28162240 0x7f7f280afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.087+0000 7ff51d7fa700 1 -- 172.21.15.113:0/1456054772 <== osd.3 v2:172.21.15.113:6808/2024634597 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff514076110 con 0x7ff514073640 2024-01-15T11:25:22.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.087+0000 7f7f2edb4700 1 --2- 172.21.15.113:0/2688521614 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f280ae4e0 0x7f7f2806c000 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.087+0000 7f7f2edb4700 1 -- 172.21.15.113:0/2688521614 >> 172.21.15.113:0/2688521614 conn(0x7f7f28071b90 msgr2=0x7f7f28071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.088+0000 7fe36affd700 1 -- 172.21.15.113:0/3569903455 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe364022070 con 0x7fe3741556a0 2024-01-15T11:25:22.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.088+0000 7fe36affd700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe35806b420 0x7fe35806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.088+0000 7fe36affd700 1 -- 172.21.15.113:0/3569903455 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe36408be60 con 0x7fe3741556a0 2024-01-15T11:25:22.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.088+0000 7fe36affd700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.141:6816/2034940459,v1:172.21.15.141:6817/2034940459] conn(0x7fe358073640 0x7fe358075a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.088+0000 7fe36affd700 1 -- 172.21.15.113:0/3569903455 --> [v2:172.21.15.141:6816/2034940459,v1:172.21.15.141:6817/2034940459] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe358076110 con 0x7fe358073640 2024-01-15T11:25:22.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.088+0000 7fe36affd700 1 -- 172.21.15.113:0/3569903455 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe36408c210 con 0x7fe3741556a0 2024-01-15T11:25:22.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.087+0000 7fa3e97b1700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:6824/2814390340,v1:172.21.15.113:6825/2814390340] conn(0x7fa3c80734f0 0x7fa3c8075910 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.089+0000 7fa3e9fb2700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa3c806b350 0x7fa3c806d810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009be0 tx=0x7fa3e0009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.089+0000 7fa3da7fc700 1 -- 172.21.15.113:0/150450123 <== osd.1 v2:172.21.15.113:6824/2814390340 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa3c8075fc0 con 0x7fa3c80734f0 2024-01-15T11:25:22.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.089+0000 7f54abfff700 1 -- 172.21.15.113:0/2844102001 <== osd.4 v2:172.21.15.141:6800/1209001813 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f54b0075fc0 con 0x7f54b00734f0 2024-01-15T11:25:22.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.090+0000 7fe37a248700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe35806b420 0x7fe35806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.090+0000 7fe37a248700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe35806b420 0x7fe35806d8e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe3741fb470 tx=0x7fe36c009250 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.090+0000 7f7f2edb4700 1 -- 172.21.15.113:0/2688521614 shutdown_connections 2024-01-15T11:25:22.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.090+0000 7f7f2edb4700 1 -- 172.21.15.113:0/2688521614 wait complete. 2024-01-15T11:25:22.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.092+0000 7f7f2edb4700 1 Processor -- start 2024-01-15T11:25:22.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.092+0000 7f4c52e00700 1 Processor -- start 2024-01-15T11:25:22.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.092+0000 7f4c52e00700 1 -- start start 2024-01-15T11:25:22.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f4c52e00700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4c4c162240 0x7f4c4c0afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f4c52e00700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f4c4c0ae4e0 0x7f4c4c06c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f4c52e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c4c06f360 0x7f4c4c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f4c52e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c0b46e0 con 0x7f4c4c06f360 2024-01-15T11:25:22.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f4c52e00700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c0b49c0 con 0x7f4c4c0ae4e0 2024-01-15T11:25:22.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f4c52e00700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0b4ca0 con 0x7f4c4c162240 2024-01-15T11:25:22.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f4c50b9c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4c4c162240 0x7f4c4c0afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f4c50b9c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4c4c162240 0x7f4c4c0afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:47352/0 (socket says 172.21.15.113:47352) 2024-01-15T11:25:22.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f4c50b9c700 1 -- 172.21.15.113:0/2577083077 learned_addr learned my addr 172.21.15.113:0/2577083077 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f4c5139d700 1 --2- 172.21.15.113:0/2577083077 >> v2:172.21.15.113:3300/0 conn(0x7f4c4c06f360 0x7f4c4c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f4c4bfff700 1 --2- 172.21.15.113:0/2577083077 >> v2:172.21.15.141:3300/0 conn(0x7f4c4c0ae4e0 0x7f4c4c06c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f4c5139d700 1 -- 172.21.15.113:0/2577083077 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4c4c162240 msgr2=0x7f4c4c0afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f4c5139d700 1 --2- 172.21.15.113:0/2577083077 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4c4c162240 0x7f4c4c0afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f4c5139d700 1 -- 172.21.15.113:0/2577083077 >> v2:172.21.15.141:3300/0 conn(0x7f4c4c0ae4e0 msgr2=0x7f4c4c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f4c5139d700 1 --2- 172.21.15.113:0/2577083077 >> v2:172.21.15.141:3300/0 conn(0x7f4c4c0ae4e0 0x7f4c4c06c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f4c5139d700 1 -- 172.21.15.113:0/2577083077 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c0b4f80 con 0x7f4c4c06f360 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.092+0000 7f7f2edb4700 1 -- start start 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f7f2edb4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f2806f360 0x7f7f2815f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f7f2edb4700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f280ae4e0 0x7f7f2814b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f7f2edb4700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7f28162240 0x7f7f28161080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f7f2edb4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f28160740 con 0x7f7f2806f360 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f7f2edb4700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f28160180 con 0x7f7f280ae4e0 2024-01-15T11:25:22.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f7f2edb4700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f28160460 con 0x7f7f28162240 2024-01-15T11:25:22.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f7f2cb50700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f2806f360 0x7f7f2815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f7f2cb50700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f2806f360 0x7f7f2815f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45626/0 (socket says 172.21.15.113:45626) 2024-01-15T11:25:22.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f7f2cb50700 1 -- 172.21.15.113:0/231852618 learned_addr learned my addr 172.21.15.113:0/231852618 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.093+0000 7f7f2d351700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7f28162240 0x7f7f28161080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f7f2d351700 1 -- 172.21.15.113:0/231852618 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f280ae4e0 msgr2=0x7f7f2814b6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T11:25:22.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f7f2d351700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7f280ae4e0 0x7f7f2814b6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f7f2d351700 1 -- 172.21.15.113:0/231852618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f2806f360 msgr2=0x7f7f2815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f7f2d351700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f2806f360 0x7f7f2815f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f7f2d351700 1 -- 172.21.15.113:0/231852618 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f14009550 con 0x7f7f28162240 2024-01-15T11:25:22.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f4c5139d700 1 --2- 172.21.15.113:0/2577083077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c06f360 0x7f4c4c06c540 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c06d380 tx=0x7f4c3c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc39cfb73e1dc72 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.095+0000 7f7f2d351700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7f28162240 0x7f7f28161080 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7f20002200 tx=0x7f7f2000cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f4c4b7fe700 1 -- 172.21.15.113:0/2577083077 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c011830 con 0x7f4c4c06f360 2024-01-15T11:25:22.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f4c4b7fe700 1 -- 172.21.15.113:0/2577083077 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c011990 con 0x7f4c4c06f360 2024-01-15T11:25:22.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.094+0000 7f4c4b7fe700 1 -- 172.21.15.113:0/2577083077 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c011d20 con 0x7f4c4c06f360 2024-01-15T11:25:22.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.095+0000 7f4c52e00700 1 -- 172.21.15.113:0/2577083077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c06f360 msgr2=0x7f4c4c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.095+0000 7f4c52e00700 1 --2- 172.21.15.113:0/2577083077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c06f360 0x7f4c4c06c540 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c06d380 tx=0x7f4c3c009b20 comp rx=0 tx=0).stop 2024-01-15T11:25:22.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.096+0000 7f4c52e00700 1 -- 172.21.15.113:0/2577083077 shutdown_connections 2024-01-15T11:25:22.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.096+0000 7f4c52e00700 1 --2- 172.21.15.113:0/2577083077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c06f360 0x7f4c4c06c540 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.096+0000 7f4c52e00700 1 --2- 172.21.15.113:0/2577083077 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4c4c162240 0x7f4c4c0afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.096+0000 7f4c52e00700 1 --2- 172.21.15.113:0/2577083077 >> v2:172.21.15.141:3300/0 conn(0x7f4c4c0ae4e0 0x7f4c4c06c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.096+0000 7f4c52e00700 1 -- 172.21.15.113:0/2577083077 >> 172.21.15.113:0/2577083077 conn(0x7f4c4c071b90 msgr2=0x7f4c4c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.098+0000 7f4c52e00700 1 -- 172.21.15.113:0/2577083077 shutdown_connections 2024-01-15T11:25:22.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.099+0000 7fe379a47700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.141:6816/2034940459,v1:172.21.15.141:6817/2034940459] conn(0x7fe358073640 0x7fe358075a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.098+0000 7f4c52e00700 1 -- 172.21.15.113:0/2577083077 wait complete. 2024-01-15T11:25:22.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.100+0000 7f4c52e00700 1 Processor -- start 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.100+0000 7f4c52e00700 1 -- start start 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.101+0000 7f4c52e00700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c4c06f360 0x7f4c4c152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.101+0000 7f4c52e00700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4c4c162240 0x7f4c4c164930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.101+0000 7f4c52e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c164e70 0x7f4c4c1ed840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.101+0000 7f4c52e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c165480 con 0x7f4c4c164e70 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.101+0000 7f4c52e00700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c165760 con 0x7f4c4c06f360 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.101+0000 7f4c52e00700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c15bac0 con 0x7f4c4c162240 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.101+0000 7f4c5139d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c164e70 0x7f4c4c1ed840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.101+0000 7f4c5139d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c164e70 0x7f4c4c1ed840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45648/0 (socket says 172.21.15.113:45648) 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.101+0000 7f4c5139d700 1 -- 172.21.15.113:0/2976139166 learned_addr learned my addr 172.21.15.113:0/2976139166 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.102+0000 7f4c4bfff700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4c4c162240 0x7f4c4c164930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.102+0000 7f4c5139d700 1 -- 172.21.15.113:0/2976139166 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4c4c162240 msgr2=0x7f4c4c164930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.102+0000 7f4c5139d700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f4c4c162240 0x7f4c4c164930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.102+0000 7f4c5139d700 1 -- 172.21.15.113:0/2976139166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c4c06f360 msgr2=0x7f4c4c152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.102+0000 7f4c5139d700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4c4c06f360 0x7f4c4c152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.102+0000 7f4c5139d700 1 -- 172.21.15.113:0/2976139166 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c0094a0 con 0x7f4c4c164e70 2024-01-15T11:25:22.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.102+0000 7f4c5139d700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c164e70 0x7f4c4c1ed840 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0728e0 tx=0x7f4c3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.102+0000 7f4c497fa700 1 -- 172.21.15.113:0/2976139166 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c0036a0 con 0x7f4c4c164e70 2024-01-15T11:25:22.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.102+0000 7f4c497fa700 1 -- 172.21.15.113:0/2976139166 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c01f070 con 0x7f4c4c164e70 2024-01-15T11:25:22.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.102+0000 7f4c497fa700 1 -- 172.21.15.113:0/2976139166 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c003a40 con 0x7f4c4c164e70 2024-01-15T11:25:22.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.102+0000 7fd91cac6700 1 -- 172.21.15.113:0/2581555752 --> [v2:172.21.15.141:6808/1489198687,v1:172.21.15.141:6809/1489198687] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd8fc005cb0 con 0x7fd8fc0015e0 2024-01-15T11:25:22.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.103+0000 7f4c52e00700 1 -- 172.21.15.113:0/2976139166 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c15bda0 con 0x7f4c4c164e70 2024-01-15T11:25:22.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.103+0000 7f4c52e00700 1 -- 172.21.15.113:0/2976139166 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c153400 con 0x7f4c4c164e70 2024-01-15T11:25:22.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.103+0000 7f4c52e00700 1 -- 172.21.15.113:0/2976139166 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c4c067f60 con 0x7f4c4c164e70 2024-01-15T11:25:22.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.105+0000 7f7f257fa700 1 -- 172.21.15.113:0/231852618 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f20003a00 con 0x7f7f28162240 2024-01-15T11:25:22.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.105+0000 7f7f257fa700 1 -- 172.21.15.113:0/231852618 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f20003b60 con 0x7f7f28162240 2024-01-15T11:25:22.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.105+0000 7f7f257fa700 1 -- 172.21.15.113:0/231852618 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2000a7b0 con 0x7f7f28162240 2024-01-15T11:25:22.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.105+0000 7f7f2edb4700 1 -- 172.21.15.113:0/231852618 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2814aee0 con 0x7f7f28162240 2024-01-15T11:25:22.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.105+0000 7f7f2edb4700 1 -- 172.21.15.113:0/231852618 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2815fd70 con 0x7f7f28162240 2024-01-15T11:25:22.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.106+0000 7f7f2edb4700 1 -- 172.21.15.113:0/231852618 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7f280b0410 con 0x7f7f28162240 2024-01-15T11:25:22.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.106+0000 7fd906ffd700 1 -- 172.21.15.113:0/2581555752 <== osd.5 v2:172.21.15.141:6808/1489198687 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd8fc005cb0 con 0x7fd8fc0015e0 2024-01-15T11:25:22.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.105+0000 7f4c497fa700 1 -- 172.21.15.113:0/2976139166 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4c3c003ba0 con 0x7f4c4c164e70 2024-01-15T11:25:22.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.105+0000 7f4c497fa700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f4c3006b450 0x7f4c3006d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.105+0000 7f4c497fa700 1 -- 172.21.15.113:0/2976139166 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c3c021070 con 0x7f4c4c164e70 2024-01-15T11:25:22.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.105+0000 7f4c497fa700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.141:6824/3801965007,v1:172.21.15.141:6825/3801965007] conn(0x7f4c30073670 0x7f4c30075a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.105+0000 7f4c497fa700 1 -- 172.21.15.113:0/2976139166 --> [v2:172.21.15.141:6824/3801965007,v1:172.21.15.141:6825/3801965007] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c30076140 con 0x7f4c30073670 2024-01-15T11:25:22.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.105+0000 7f4c497fa700 1 -- 172.21.15.113:0/2976139166 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c3c08bd10 con 0x7f4c4c164e70 2024-01-15T11:25:22.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.106+0000 7f4c4bfff700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.141:6824/3801965007,v1:172.21.15.141:6825/3801965007] conn(0x7f4c30073670 0x7f4c30075a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.106+0000 7fd91cac6700 1 -- 172.21.15.113:0/2581555752 >> [v2:172.21.15.141:6808/1489198687,v1:172.21.15.141:6809/1489198687] conn(0x7fd8fc0015e0 msgr2=0x7fd8fc003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.106+0000 7fd91cac6700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.141:6808/1489198687,v1:172.21.15.141:6809/1489198687] conn(0x7fd8fc0015e0 0x7fd8fc003aa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.106+0000 7f4c4bfff700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.141:6824/3801965007,v1:172.21.15.141:6825/3801965007] conn(0x7f4c30073670 0x7f4c30075a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.106+0000 7fd91cac6700 1 -- 172.21.15.113:0/2581555752 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd8f806b3a0 msgr2=0x7fd8f806d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.106+0000 7fd91cac6700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd8f806b3a0 0x7fd8f806d860 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009a10 tx=0x7fd90c0091c0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.106+0000 7fd91cac6700 1 -- 172.21.15.113:0/2581555752 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd91814ba00 msgr2=0x7fd9181617f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.106+0000 7fd91cac6700 1 --2- 172.21.15.113:0/2581555752 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd91814ba00 0x7fd9181617f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd910014000 tx=0x7fd910004ad0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.107+0000 7f4c50b9c700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f4c3006b450 0x7f4c3006d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.107+0000 7f7f257fa700 1 -- 172.21.15.113:0/231852618 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7f2000a910 con 0x7f7f28162240 2024-01-15T11:25:22.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.108+0000 7f7f257fa700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7f0c06b420 0x7f7f0c06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.108+0000 7f7f2cb50700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7f0c06b420 0x7f7f0c06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.108+0000 7f7f257fa700 1 -- 172.21.15.113:0/231852618 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f200ac330 con 0x7f7f28162240 2024-01-15T11:25:22.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.108+0000 7f7f257fa700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.113:6800/1061671788,v1:172.21.15.113:6801/1061671788] conn(0x7f7f0c073640 0x7f7f0c075a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.108+0000 7f7f257fa700 1 -- 172.21.15.113:0/231852618 --> [v2:172.21.15.113:6800/1061671788,v1:172.21.15.113:6801/1061671788] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f0c076110 con 0x7f7f0c073640 2024-01-15T11:25:22.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.108+0000 7f7f257fa700 1 -- 172.21.15.113:0/231852618 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f200ac6e0 con 0x7f7f28162240 2024-01-15T11:25:22.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.109+0000 7f7f2cb50700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7f0c06b420 0x7f7f0c06d8e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7f2809ef80 tx=0x7f7f1c009290 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.109+0000 7f7f27fff700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.113:6800/1061671788,v1:172.21.15.113:6801/1061671788] conn(0x7f7f0c073640 0x7f7f0c075a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.107+0000 7f436f114700 1 -- 172.21.15.113:0/2058816143 --> [v2:172.21.15.113:6816/1186622624,v1:172.21.15.113:6817/1186622624] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4348002d40 con 0x7f4354073640 2024-01-15T11:25:22.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.109+0000 7f43657fa700 1 -- 172.21.15.113:0/2058816143 <== osd.2 v2:172.21.15.113:6816/1186622624 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4348002d40 con 0x7f4354073640 2024-01-15T11:25:22.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7f436f114700 1 -- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:6816/1186622624,v1:172.21.15.113:6817/1186622624] conn(0x7f4354073640 msgr2=0x7f4354075a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7f436f114700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:6816/1186622624,v1:172.21.15.113:6817/1186622624] conn(0x7f4354073640 0x7f4354075a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7f436f114700 1 -- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f435406b420 msgr2=0x7f435406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7f436f114700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f435406b420 0x7f435406d8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4358005910 tx=0x7f4358009040 comp rx=0 tx=0).stop 2024-01-15T11:25:22.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7f436f114700 1 -- 172.21.15.113:0/2058816143 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f43680b4260 msgr2=0x7f43681ea200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7f436f114700 1 --2- 172.21.15.113:0/2058816143 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f43680b4260 0x7f43681ea200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f436001e000 tx=0x7f4360004ad0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7f436d6b1700 1 -- 172.21.15.113:0/2058816143 reap_dead start 2024-01-15T11:25:22.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7f436f114700 1 -- 172.21.15.113:0/2058816143 shutdown_connections 2024-01-15T11:25:22.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7f436f114700 1 -- 172.21.15.113:0/2058816143 >> 172.21.15.113:0/2058816143 conn(0x7f43680cfa30 msgr2=0x7f436815be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7f436f114700 1 -- 172.21.15.113:0/2058816143 shutdown_connections 2024-01-15T11:25:22.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7f436f114700 1 -- 172.21.15.113:0/2058816143 wait complete. 2024-01-15T11:25:22.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7fd916d9d700 1 -- 172.21.15.113:0/2581555752 reap_dead start 2024-01-15T11:25:22.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.107+0000 7f4c497fa700 1 -- 172.21.15.113:0/2976139166 <== osd.6 v2:172.21.15.141:6824/3801965007 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4c30076140 con 0x7f4c30073670 2024-01-15T11:25:22.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.107+0000 7f4c50b9c700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f4c3006b450 0x7f4c3006d910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009fa0 tx=0x7f4c4400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.108+0000 7fe379a47700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.141:6816/2034940459,v1:172.21.15.141:6817/2034940459] conn(0x7fe358073640 0x7fe358075a60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.108+0000 7fe36affd700 1 -- 172.21.15.113:0/3569903455 <== osd.7 v2:172.21.15.141:6816/2034940459 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe358076110 con 0x7fe358073640 2024-01-15T11:25:22.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.110+0000 7f7f27fff700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.113:6800/1061671788,v1:172.21.15.113:6801/1061671788] conn(0x7f7f0c073640 0x7f7f0c075a60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.112+0000 7f7f257fa700 1 -- 172.21.15.113:0/231852618 <== osd.0 v2:172.21.15.113:6800/1061671788 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7f0c076110 con 0x7f7f0c073640 2024-01-15T11:25:22.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.117+0000 7fd91cac6700 1 -- 172.21.15.113:0/2581555752 shutdown_connections 2024-01-15T11:25:22.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.117+0000 7fd91cac6700 1 -- 172.21.15.113:0/2581555752 >> 172.21.15.113:0/2581555752 conn(0x7fd9180731a0 msgr2=0x7fd918163f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.117+0000 7fd91cac6700 1 -- 172.21.15.113:0/2581555752 shutdown_connections 2024-01-15T11:25:22.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.117+0000 7fd91cac6700 1 -- 172.21.15.113:0/2581555752 wait complete. 2024-01-15T11:25:22.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.122+0000 7ff52eff6700 1 -- 172.21.15.113:0/1456054772 --> [v2:172.21.15.113:6808/2024634597,v1:172.21.15.113:6809/2024634597] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff528067f60 con 0x7ff514073640 2024-01-15T11:25:22.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.123+0000 7ff51d7fa700 1 -- 172.21.15.113:0/1456054772 <== osd.3 v2:172.21.15.113:6808/2024634597 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff528067f60 con 0x7ff514073640 2024-01-15T11:25:22.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.123+0000 7ff52eff6700 1 -- 172.21.15.113:0/1456054772 >> [v2:172.21.15.113:6808/2024634597,v1:172.21.15.113:6809/2024634597] conn(0x7ff514073640 msgr2=0x7ff514075a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.123+0000 7ff52eff6700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.113:6808/2024634597,v1:172.21.15.113:6809/2024634597] conn(0x7ff514073640 0x7ff514075a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.124+0000 7fa3ec216700 1 -- 172.21.15.113:0/150450123 --> [v2:172.21.15.113:6824/2814390340,v1:172.21.15.113:6825/2814390340] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa3e4152580 con 0x7fa3c80734f0 2024-01-15T11:25:22.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.125+0000 7fa3da7fc700 1 -- 172.21.15.113:0/150450123 <== osd.1 v2:172.21.15.113:6824/2814390340 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa3e4152580 con 0x7fa3c80734f0 2024-01-15T11:25:22.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.132+0000 7f54c9f10700 1 -- 172.21.15.113:0/2844102001 --> [v2:172.21.15.141:6800/1209001813,v1:172.21.15.141:6801/1209001813] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f54c40b3150 con 0x7f54b00734f0 2024-01-15T11:25:22.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.135+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3569903455 --> [v2:172.21.15.141:6816/2034940459,v1:172.21.15.141:6817/2034940459] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe35c002d40 con 0x7fe358073640 2024-01-15T11:25:22.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.135+0000 7ff52eff6700 1 -- 172.21.15.113:0/1456054772 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff51406b420 msgr2=0x7ff51406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.135+0000 7ff52eff6700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff51406b420 0x7ff51406d8e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff524009a30 tx=0x7ff524009190 comp rx=0 tx=0).stop 2024-01-15T11:25:22.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.135+0000 7ff52eff6700 1 -- 172.21.15.113:0/1456054772 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5281f6ee0 msgr2=0x7ff5280719a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.135+0000 7ff52eff6700 1 --2- 172.21.15.113:0/1456054772 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff5281f6ee0 0x7ff5280719a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff52000ef90 tx=0x7ff52000c3a0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.135+0000 7ff52d593700 1 -- 172.21.15.113:0/1456054772 reap_dead start 2024-01-15T11:25:22.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.135+0000 7ff52eff6700 1 -- 172.21.15.113:0/1456054772 shutdown_connections 2024-01-15T11:25:22.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.135+0000 7ff52eff6700 1 -- 172.21.15.113:0/1456054772 >> 172.21.15.113:0/1456054772 conn(0x7ff5280d3ed0 msgr2=0x7ff5280d2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.135+0000 7fe36affd700 1 -- 172.21.15.113:0/3569903455 <== osd.7 v2:172.21.15.141:6816/2034940459 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe35c002d40 con 0x7fe358073640 2024-01-15T11:25:22.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.135+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3569903455 >> [v2:172.21.15.141:6816/2034940459,v1:172.21.15.141:6817/2034940459] conn(0x7fe358073640 msgr2=0x7fe358075a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.135+0000 7fe37c4ac700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.141:6816/2034940459,v1:172.21.15.141:6817/2034940459] conn(0x7fe358073640 0x7fe358075a60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.136+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3569903455 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe35806b420 msgr2=0x7fe35806d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.136+0000 7fe37c4ac700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe35806b420 0x7fe35806d8e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe3741fb470 tx=0x7fe36c009250 comp rx=0 tx=0).stop 2024-01-15T11:25:22.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.136+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3569903455 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe3741556a0 msgr2=0x7fe37413f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.136+0000 7fe37c4ac700 1 --2- 172.21.15.113:0/3569903455 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe3741556a0 0x7fe37413f580 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe3740737d0 tx=0x7fe364004750 comp rx=0 tx=0).stop 2024-01-15T11:25:22.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.136+0000 7f54abfff700 1 -- 172.21.15.113:0/2844102001 <== osd.4 v2:172.21.15.141:6800/1209001813 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f54b8001470 con 0x7f54b00734f0 2024-01-15T11:25:22.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.137+0000 7ff52eff6700 1 -- 172.21.15.113:0/1456054772 shutdown_connections 2024-01-15T11:25:22.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.137+0000 7ff52eff6700 1 -- 172.21.15.113:0/1456054772 wait complete. 2024-01-15T11:25:22.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.137+0000 7f54c9f10700 1 -- 172.21.15.113:0/2844102001 >> [v2:172.21.15.141:6800/1209001813,v1:172.21.15.141:6801/1209001813] conn(0x7f54b00734f0 msgr2=0x7f54b0075910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.137+0000 7f54c9f10700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.141:6800/1209001813,v1:172.21.15.141:6801/1209001813] conn(0x7f54b00734f0 0x7f54b0075910 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.137+0000 7f54c9f10700 1 -- 172.21.15.113:0/2844102001 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f54b006b350 msgr2=0x7f54b006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.137+0000 7f54c9f10700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f54b006b350 0x7f54b006d810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f54b40097c0 tx=0x7f54b401c040 comp rx=0 tx=0).stop 2024-01-15T11:25:22.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.137+0000 7f54c9f10700 1 -- 172.21.15.113:0/2844102001 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54c41665a0 msgr2=0x7f54c4200080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.137+0000 7f54c9f10700 1 --2- 172.21.15.113:0/2844102001 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f54c41665a0 0x7f54c4200080 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f54c40d0ae0 tx=0x7f54bc00cd80 comp rx=0 tx=0).stop 2024-01-15T11:25:22.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.137+0000 7f54c3fff700 1 -- 172.21.15.113:0/2844102001 reap_dead start 2024-01-15T11:25:22.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.137+0000 7f54c9f10700 1 -- 172.21.15.113:0/2844102001 shutdown_connections 2024-01-15T11:25:22.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.137+0000 7f54c9f10700 1 -- 172.21.15.113:0/2844102001 >> 172.21.15.113:0/2844102001 conn(0x7f54c40b1740 msgr2=0x7f54c40b0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.137+0000 7f54c9f10700 1 -- 172.21.15.113:0/2844102001 shutdown_connections 2024-01-15T11:25:22.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7fa3ec216700 1 -- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:6824/2814390340,v1:172.21.15.113:6825/2814390340] conn(0x7fa3c80734f0 msgr2=0x7fa3c8075910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7fa3ec216700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:6824/2814390340,v1:172.21.15.113:6825/2814390340] conn(0x7fa3c80734f0 0x7fa3c8075910 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7fa3ec216700 1 -- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa3c806b350 msgr2=0x7fa3c806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7fa3ec216700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa3c806b350 0x7fa3c806d810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009be0 tx=0x7fa3e0009210 comp rx=0 tx=0).stop 2024-01-15T11:25:22.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7fa3ec216700 1 -- 172.21.15.113:0/150450123 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3e406c210 msgr2=0x7fa3e4154ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7fa3ec216700 1 --2- 172.21.15.113:0/150450123 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa3e406c210 0x7fa3e4154ae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc015000 tx=0x7fa3dc004ad0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7fa3ea7b3700 1 -- 172.21.15.113:0/150450123 reap_dead start 2024-01-15T11:25:22.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7fa3ec216700 1 -- 172.21.15.113:0/150450123 shutdown_connections 2024-01-15T11:25:22.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7fa3ec216700 1 -- 172.21.15.113:0/150450123 >> 172.21.15.113:0/150450123 conn(0x7fa3e40d3ed0 msgr2=0x7fa3e40d21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7fa3ec216700 1 -- 172.21.15.113:0/150450123 shutdown_connections 2024-01-15T11:25:22.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7fa3ec216700 1 -- 172.21.15.113:0/150450123 wait complete. 2024-01-15T11:25:22.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7f4c52e00700 1 -- 172.21.15.113:0/2976139166 --> [v2:172.21.15.141:6824/3801965007,v1:172.21.15.141:6825/3801965007] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4c4c15c2c0 con 0x7f4c30073670 2024-01-15T11:25:22.140 INFO:teuthology.orchestra.run.smithi113.stdout:55834574855 2024-01-15T11:25:22.140 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-15T11:25:22.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7f54c9f10700 1 -- 172.21.15.113:0/2844102001 wait complete. 2024-01-15T11:25:22.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.138+0000 7fe37aa49700 1 -- 172.21.15.113:0/3569903455 reap_dead start 2024-01-15T11:25:22.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.139+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3569903455 shutdown_connections 2024-01-15T11:25:22.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.139+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3569903455 >> 172.21.15.113:0/3569903455 conn(0x7fe3740cfa30 msgr2=0x7fe3740b3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.139+0000 7f4c497fa700 1 -- 172.21.15.113:0/2976139166 <== osd.6 v2:172.21.15.141:6824/3801965007 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4c4c15c2c0 con 0x7f4c30073670 2024-01-15T11:25:22.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.139+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3569903455 shutdown_connections 2024-01-15T11:25:22.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.139+0000 7fe37c4ac700 1 -- 172.21.15.113:0/3569903455 wait complete. 2024-01-15T11:25:22.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.139+0000 7f4c52e00700 1 -- 172.21.15.113:0/2976139166 >> [v2:172.21.15.141:6824/3801965007,v1:172.21.15.141:6825/3801965007] conn(0x7f4c30073670 msgr2=0x7f4c30075a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.139+0000 7f4c52e00700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.141:6824/3801965007,v1:172.21.15.141:6825/3801965007] conn(0x7f4c30073670 0x7f4c30075a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.139+0000 7f4c52e00700 1 -- 172.21.15.113:0/2976139166 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f4c3006b450 msgr2=0x7f4c3006d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.139+0000 7f4c52e00700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f4c3006b450 0x7f4c3006d910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009fa0 tx=0x7f4c4400b040 comp rx=0 tx=0).stop 2024-01-15T11:25:22.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.139+0000 7f4c52e00700 1 -- 172.21.15.113:0/2976139166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c164e70 msgr2=0x7f4c4c1ed840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.139+0000 7f4c52e00700 1 --2- 172.21.15.113:0/2976139166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c164e70 0x7f4c4c1ed840 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0728e0 tx=0x7f4c3c004750 comp rx=0 tx=0).stop 2024-01-15T11:25:22.143 INFO:teuthology.orchestra.run.smithi113.stdout:55834574855 2024-01-15T11:25:22.143 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-15T11:25:22.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.142+0000 7f4c5139d700 1 -- 172.21.15.113:0/2976139166 reap_dead start 2024-01-15T11:25:22.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.142+0000 7f4c52e00700 1 -- 172.21.15.113:0/2976139166 shutdown_connections 2024-01-15T11:25:22.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.142+0000 7f4c52e00700 1 -- 172.21.15.113:0/2976139166 >> 172.21.15.113:0/2976139166 conn(0x7f4c4c071b90 msgr2=0x7f4c4c06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.142+0000 7f4c52e00700 1 -- 172.21.15.113:0/2976139166 shutdown_connections 2024-01-15T11:25:22.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.142+0000 7f4c52e00700 1 -- 172.21.15.113:0/2976139166 wait complete. 2024-01-15T11:25:22.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.144+0000 7f7f2edb4700 1 -- 172.21.15.113:0/231852618 --> [v2:172.21.15.113:6800/1061671788,v1:172.21.15.113:6801/1061671788] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7f280b5260 con 0x7f7f0c073640 2024-01-15T11:25:22.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.145+0000 7f7f257fa700 1 -- 172.21.15.113:0/231852618 <== osd.0 v2:172.21.15.113:6800/1061671788 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7f14006c60 con 0x7f7f0c073640 2024-01-15T11:25:22.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.146+0000 7f7f2edb4700 1 -- 172.21.15.113:0/231852618 >> [v2:172.21.15.113:6800/1061671788,v1:172.21.15.113:6801/1061671788] conn(0x7f7f0c073640 msgr2=0x7f7f0c075a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.146+0000 7f7f2edb4700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.113:6800/1061671788,v1:172.21.15.113:6801/1061671788] conn(0x7f7f0c073640 0x7f7f0c075a60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.146+0000 7f7f2edb4700 1 -- 172.21.15.113:0/231852618 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7f0c06b420 msgr2=0x7f7f0c06d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.146+0000 7f7f2edb4700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7f0c06b420 0x7f7f0c06d8e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7f2809ef80 tx=0x7f7f1c009290 comp rx=0 tx=0).stop 2024-01-15T11:25:22.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.146+0000 7f7f2edb4700 1 -- 172.21.15.113:0/231852618 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7f28162240 msgr2=0x7f7f28161080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.146+0000 7f7f2edb4700 1 --2- 172.21.15.113:0/231852618 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7f28162240 0x7f7f28161080 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7f20002200 tx=0x7f7f2000cd80 comp rx=0 tx=0).stop 2024-01-15T11:25:22.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.146+0000 7f7f2d351700 1 -- 172.21.15.113:0/231852618 reap_dead start 2024-01-15T11:25:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.151+0000 7f7f2edb4700 1 -- 172.21.15.113:0/231852618 shutdown_connections 2024-01-15T11:25:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.151+0000 7f7f2edb4700 1 -- 172.21.15.113:0/231852618 >> 172.21.15.113:0/231852618 conn(0x7f7f28071b90 msgr2=0x7f7f2806b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.151+0000 7f7f2edb4700 1 -- 172.21.15.113:0/231852618 shutdown_connections 2024-01-15T11:25:22.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.151+0000 7f7f2edb4700 1 -- 172.21.15.113:0/231852618 wait complete. 2024-01-15T11:25:22.164 INFO:teuthology.orchestra.run.smithi113.stdout:55834574855 2024-01-15T11:25:22.164 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-15T11:25:22.165 INFO:teuthology.orchestra.run.smithi113.stdout:55834574857 2024-01-15T11:25:22.165 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-15T11:25:22.166 INFO:teuthology.orchestra.run.smithi113.stdout:55834574856 2024-01-15T11:25:22.166 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-15T11:25:22.167 INFO:teuthology.orchestra.run.smithi113.stdout:55834574854 2024-01-15T11:25:22.167 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-15T11:25:22.168 INFO:teuthology.orchestra.run.smithi113.stdout:55834574856 2024-01-15T11:25:22.168 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-15T11:25:22.171 INFO:teuthology.orchestra.run.smithi113.stdout:55834574854 2024-01-15T11:25:22.172 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-15T11:25:22.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.364+0000 7fe547fff700 1 Processor -- start 2024-01-15T11:25:22.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.364+0000 7fe547fff700 1 -- start start 2024-01-15T11:25:22.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.366+0000 7fe547fff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe5480cc2d0 0x7fe54815af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.366+0000 7fe547fff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe54815e550 0x7fe54806b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.366+0000 7fe547fff700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5481553c0 con 0x7fe548154fe0 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.366+0000 7fe547fff700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54806b190 con 0x7fe5480cc2d0 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.366+0000 7fe547fff700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54815b540 con 0x7fe54815e550 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.367+0000 7fe546ffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fe548154fe0 0x7fe5480cbbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:37504/0 (socket says 172.21.15.113:37504) 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.367+0000 7fe546ffd700 1 -- 172.21.15.113:0/2868766817 learned_addr learned my addr 172.21.15.113:0/2868766817 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe5477fe700 1 --2- 172.21.15.113:0/2868766817 >> v2:172.21.15.141:3300/0 conn(0x7fe54815e550 0x7fe54806b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134426990 0 0) 0x7fe5481553c0 con 0x7fe548154fe0 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5300035b0 con 0x7fe548154fe0 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1046687727 0 0) 0x7fe5300035b0 con 0x7fe548154fe0 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 >> v2:172.21.15.141:3300/0 conn(0x7fe54815e550 msgr2=0x7fe54806b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe545ffb700 1 --2- 172.21.15.113:0/2868766817 >> v2:172.21.15.141:3300/0 conn(0x7fe54815e550 0x7fe54806b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 >> v2:172.21.15.113:3300/0 conn(0x7fe5480cc2d0 msgr2=0x7fe54815af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe545ffb700 1 --2- 172.21.15.113:0/2868766817 >> v2:172.21.15.113:3300/0 conn(0x7fe5480cc2d0 0x7fe54815af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54815b820 con 0x7fe548154fe0 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fe538001670 con 0x7fe548154fe0 2024-01-15T11:25:22.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 >> v1:172.21.15.113:6789/0 conn(0x7fe548154fe0 legacy=0x7fe5480cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe545ffb700 1 --2- 172.21.15.113:0/2868766817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe530003a00 0x7fe530003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.368+0000 7fe545ffb700 1 --2- 172.21.15.113:0/2868766817 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe530004850 0x7fe530004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7fe545ffb700 1 --2- 172.21.15.113:0/2868766817 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 0x7fe53002daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54806b190 con 0x7fe530003a00 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54815b540 con 0x7fe530004850 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53002e400 con 0x7fe530005460 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7fe5477fe700 1 --2- 172.21.15.113:0/2868766817 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 0x7fe53002daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7fe5467fc700 1 --2- 172.21.15.113:0/2868766817 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe530004850 0x7fe530004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7fe5477fe700 1 -- 172.21.15.113:0/2868766817 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe530004850 msgr2=0x7fe530004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7fe5477fe700 1 --2- 172.21.15.113:0/2868766817 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe530004850 0x7fe530004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7fe5477fe700 1 -- 172.21.15.113:0/2868766817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe530003a00 msgr2=0x7fe530003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7fe5477fe700 1 --2- 172.21.15.113:0/2868766817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe530003a00 0x7fe530003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7fe5477fe700 1 -- 172.21.15.113:0/2868766817 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54815b820 con 0x7fe530005460 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7fe5477fe700 1 -- 172.21.15.113:0/2868766817 reap_dead start 2024-01-15T11:25:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.370+0000 7fe5477fe700 1 --2- 172.21.15.113:0/2868766817 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 0x7fe53002daa0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe540007f60 tx=0x7fe530005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86580283debc59db server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.367+0000 7f6092776700 1 Processor -- start 2024-01-15T11:25:22.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.367+0000 7f6092776700 1 -- start start 2024-01-15T11:25:22.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7f6092776700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f608c071470 0x7f608c0b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7f6092776700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608c06d550 con 0x7f608c140680 2024-01-15T11:25:22.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7f6092776700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608c06cc60 con 0x7f608c0b3d40 2024-01-15T11:25:22.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7f6092776700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f608c0b47c0 con 0x7f608c071470 2024-01-15T11:25:22.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7f6090d13700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f608c071470 0x7f608c0b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7f6090d13700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f608c071470 0x7f608c0b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51398/0 (socket says 172.21.15.113:51398) 2024-01-15T11:25:22.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.369+0000 7f6090d13700 1 -- 172.21.15.113:0/1100764166 learned_addr learned my addr 172.21.15.113:0/1100764166 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.370+0000 7f6090d13700 1 -- 172.21.15.113:0/1100764166 >> v1:172.21.15.141:6789/0 conn(0x7f608c0b3d40 legacy=0x7f608c070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.370+0000 7f6090d13700 1 -- 172.21.15.113:0/1100764166 >> v1:172.21.15.113:6789/0 conn(0x7f608c140680 legacy=0x7f608c140c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.370+0000 7f6090d13700 1 -- 172.21.15.113:0/1100764166 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608c06cf00 con 0x7f608c071470 2024-01-15T11:25:22.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.370+0000 7f6090d13700 1 --2- 172.21.15.113:0/1100764166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f608c071470 0x7f608c0b41d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6084009e80 tx=0x7f608400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9299f8d32b1070f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.370+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5400079d0 con 0x7fe530005460 2024-01-15T11:25:22.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.370+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54000cde0 con 0x7fe530005460 2024-01-15T11:25:22.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.370+0000 7fe545ffb700 1 -- 172.21.15.113:0/2868766817 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe540003c40 con 0x7fe530005460 2024-01-15T11:25:22.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.370+0000 7f608affd700 1 -- 172.21.15.113:0/1100764166 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608400e070 con 0x7f608c071470 2024-01-15T11:25:22.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.371+0000 7f608affd700 1 -- 172.21.15.113:0/1100764166 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6084003a20 con 0x7f608c071470 2024-01-15T11:25:22.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.371+0000 7f608affd700 1 -- 172.21.15.113:0/1100764166 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6084003db0 con 0x7f608c071470 2024-01-15T11:25:22.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.371+0000 7fe547fff700 1 -- 172.21.15.113:0/2868766817 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 msgr2=0x7fe53002daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.371+0000 7fe547fff700 1 --2- 172.21.15.113:0/2868766817 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 0x7fe53002daa0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe540007f60 tx=0x7fe530005880 comp rx=0 tx=0).stop 2024-01-15T11:25:22.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.372+0000 7f6092776700 1 -- 172.21.15.113:0/1100764166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f608c071470 msgr2=0x7f608c0b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.372+0000 7f6092776700 1 --2- 172.21.15.113:0/1100764166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f608c071470 0x7f608c0b41d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6084009e80 tx=0x7f608400b330 comp rx=0 tx=0).stop 2024-01-15T11:25:22.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.372+0000 7fe547fff700 1 -- 172.21.15.113:0/2868766817 shutdown_connections 2024-01-15T11:25:22.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.372+0000 7fe547fff700 1 --2- 172.21.15.113:0/2868766817 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 0x7fe53002daa0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.372+0000 7fe547fff700 1 -- 172.21.15.113:0/2868766817 >> 172.21.15.113:0/2868766817 conn(0x7fe5480cd340 msgr2=0x7fe5480cd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.372+0000 7fe547fff700 1 -- 172.21.15.113:0/2868766817 shutdown_connections 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.372+0000 7fe547fff700 1 -- 172.21.15.113:0/2868766817 wait complete. 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.374+0000 7fe547fff700 1 Processor -- start 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.374+0000 7fe547fff700 1 -- start start 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.375+0000 7fe547fff700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 0x7fe54808aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.375+0000 7fe547fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5480b5e20 0x7fe548142670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.375+0000 7fe547fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe548142bb0 0x7fe548089a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.375+0000 7fe547fff700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480b62a0 con 0x7fe5480b5e20 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.375+0000 7fe547fff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe548143040 con 0x7fe548142bb0 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.375+0000 7fe547fff700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54813fad0 con 0x7fe530005460 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7fe546ffd700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 0x7fe54808aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7fe5477fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe548142bb0 0x7fe548089a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7fe546ffd700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 0x7fe54808aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:47374/0 (socket says 172.21.15.113:47374) 2024-01-15T11:25:22.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7fe546ffd700 1 -- 172.21.15.113:0/3329661122 learned_addr learned my addr 172.21.15.113:0/3329661122 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7fe5467fc700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5480b5e20 0x7fe548142670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7fe546ffd700 1 -- 172.21.15.113:0/3329661122 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe548142bb0 msgr2=0x7fe548089a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7fe546ffd700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe548142bb0 0x7fe548089a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7fe546ffd700 1 -- 172.21.15.113:0/3329661122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5480b5e20 msgr2=0x7fe548142670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7fe546ffd700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5480b5e20 0x7fe548142670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7fe546ffd700 1 -- 172.21.15.113:0/3329661122 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe548086590 con 0x7fe530005460 2024-01-15T11:25:22.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7fe5467fc700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5480b5e20 0x7fe548142670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:22.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7f6092776700 1 -- 172.21.15.113:0/1100764166 shutdown_connections 2024-01-15T11:25:22.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7f6092776700 1 --2- 172.21.15.113:0/1100764166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f608c071470 0x7f608c0b41d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.376+0000 7f6092776700 1 -- 172.21.15.113:0/1100764166 >> 172.21.15.113:0/1100764166 conn(0x7f608c0b1740 msgr2=0x7f608c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.377+0000 7fe546ffd700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 0x7fe54808aba0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe53800caf0 tx=0x7fe53800ce00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.377+0000 7f6092776700 1 -- 172.21.15.113:0/1100764166 shutdown_connections 2024-01-15T11:25:22.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.377+0000 7f6092776700 1 -- 172.21.15.113:0/1100764166 wait complete. 2024-01-15T11:25:22.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.377+0000 7fe52f7fe700 1 -- 172.21.15.113:0/3329661122 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538007360 con 0x7fe530005460 2024-01-15T11:25:22.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.377+0000 7fe52f7fe700 1 -- 172.21.15.113:0/3329661122 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5380074c0 con 0x7fe530005460 2024-01-15T11:25:22.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.377+0000 7fe52f7fe700 1 -- 172.21.15.113:0/3329661122 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538004870 con 0x7fe530005460 2024-01-15T11:25:22.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.378+0000 7f6092776700 1 Processor -- start 2024-01-15T11:25:22.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.379+0000 7f6092776700 1 -- start start 2024-01-15T11:25:22.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.378+0000 7fe547fff700 1 -- 172.21.15.113:0/3329661122 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe548086800 con 0x7fe530005460 2024-01-15T11:25:22.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.378+0000 7fe547fff700 1 -- 172.21.15.113:0/3329661122 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe548086dc0 con 0x7fe530005460 2024-01-15T11:25:22.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.379+0000 7fe547fff700 1 -- 172.21.15.113:0/3329661122 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe548086fe0 con 0x7fe530005460 2024-01-15T11:25:22.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.380+0000 7f6092776700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f608c071470 0x7f608c14ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.380+0000 7f6092776700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f608c0b3d40 0x7f608c0c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.380+0000 7f6092776700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f608c140680 0x7f608c0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.380+0000 7f6092776700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c14bf80 con 0x7f608c0b3d40 2024-01-15T11:25:22.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.380+0000 7f6092776700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c14cdc0 con 0x7f608c140680 2024-01-15T11:25:22.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.380+0000 7f6092776700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608c14ef60 con 0x7f608c071470 2024-01-15T11:25:22.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.380+0000 7f608b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f608c0b3d40 0x7f608c0c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.380+0000 7f608b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f608c0b3d40 0x7f608c0c49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45680/0 (socket says 172.21.15.113:45680) 2024-01-15T11:25:22.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.380+0000 7f608b7fe700 1 -- 172.21.15.113:0/1048633126 learned_addr learned my addr 172.21.15.113:0/1048633126 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.380+0000 7f608bfff700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f608c071470 0x7f608c14ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.381+0000 7f608b7fe700 1 -- 172.21.15.113:0/1048633126 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f608c071470 msgr2=0x7f608c14ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.381+0000 7f608b7fe700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f608c071470 0x7f608c14ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.381+0000 7f608b7fe700 1 -- 172.21.15.113:0/1048633126 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f608c140680 msgr2=0x7f608c0b0400 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T11:25:22.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.381+0000 7f608b7fe700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f608c140680 0x7f608c0b0400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.381+0000 7f608b7fe700 1 -- 172.21.15.113:0/1048633126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6084009b30 con 0x7f608c0b3d40 2024-01-15T11:25:22.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.381+0000 7fe52f7fe700 1 -- 172.21.15.113:0/3329661122 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe5380049d0 con 0x7fe530005460 2024-01-15T11:25:22.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.381+0000 7f608b7fe700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f608c0b3d40 0x7f608c0c49b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f608000c680 tx=0x7f608000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.381+0000 7f34b18b7700 1 Processor -- start 2024-01-15T11:25:22.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.382+0000 7f34b18b7700 1 -- start start 2024-01-15T11:25:22.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.382+0000 7fe52f7fe700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe53406b420 0x7fe53406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.382+0000 7fe52f7fe700 1 -- 172.21.15.113:0/3329661122 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5380895c0 con 0x7fe530005460 2024-01-15T11:25:22.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.382+0000 7f9ce2625700 1 Processor -- start 2024-01-15T11:25:22.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7fe5467fc700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe53406b420 0x7fe53406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.382+0000 7f9ce2625700 1 -- start start 2024-01-15T11:25:22.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.382+0000 7f6088ff9700 1 -- 172.21.15.113:0/1048633126 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60800054e0 con 0x7f608c0b3d40 2024-01-15T11:25:22.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.382+0000 7f6088ff9700 1 -- 172.21.15.113:0/1048633126 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6080003680 con 0x7f608c0b3d40 2024-01-15T11:25:22.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.382+0000 7f6088ff9700 1 -- 172.21.15.113:0/1048633126 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080003a30 con 0x7f608c0b3d40 2024-01-15T11:25:22.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.382+0000 7f6092776700 1 -- 172.21.15.113:0/1048633126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608c14c1b0 con 0x7f608c0b3d40 2024-01-15T11:25:22.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.382+0000 7f6092776700 1 -- 172.21.15.113:0/1048633126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f608c14c770 con 0x7f608c0b3d40 2024-01-15T11:25:22.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f34b18b7700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f34ac152cb0 0x7f34ac1601d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f34b18b7700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f34ac0ae6c0 0x7f34ac0aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f34b18b7700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac0af050 con 0x7f34ac15c1e0 2024-01-15T11:25:22.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f34b18b7700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34ac0b2b90 con 0x7f34ac0ae6c0 2024-01-15T11:25:22.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f34b18b7700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac160710 con 0x7f34ac152cb0 2024-01-15T11:25:22.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7fe5467fc700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe53406b420 0x7fe53406d8e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00ba70 tx=0x7fe53c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f6092776700 1 -- 172.21.15.113:0/1048633126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c0052f0 con 0x7f608c0b3d40 2024-01-15T11:25:22.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f34abfff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f34ac15c1e0 0x7f34ac15d220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:37520/0 (socket says 172.21.15.113:37520) 2024-01-15T11:25:22.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f34abfff700 1 -- 172.21.15.113:0/838835374 learned_addr learned my addr 172.21.15.113:0/838835374 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:22.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f34b08b5700 1 --2- 172.21.15.113:0/838835374 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f34ac152cb0 0x7f34ac1601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f34b10b6700 1 --2- 172.21.15.113:0/838835374 >> v2:172.21.15.141:3300/0 conn(0x7f34ac0ae6c0 0x7f34ac0aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f34ab7fe700 1 -- 172.21.15.113:0/838835374 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558123328 0 0) 0x7f3494001110 con 0x7f34ac15c1e0 2024-01-15T11:25:22.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f34ab7fe700 1 -- 172.21.15.113:0/838835374 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34980035b0 con 0x7f34ac15c1e0 2024-01-15T11:25:22.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f34b08b5700 1 -- 172.21.15.113:0/838835374 >> v2:172.21.15.141:3300/0 conn(0x7f34ac0ae6c0 msgr2=0x7f34ac0aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f34b08b5700 1 --2- 172.21.15.113:0/838835374 >> v2:172.21.15.141:3300/0 conn(0x7f34ac0ae6c0 0x7f34ac0aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f34b08b5700 1 -- 172.21.15.113:0/838835374 >> v1:172.21.15.113:6789/0 conn(0x7f34ac15c1e0 legacy=0x7f34ac15d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f34b08b5700 1 -- 172.21.15.113:0/838835374 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac153090 con 0x7f34ac152cb0 2024-01-15T11:25:22.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f34b08b5700 1 --2- 172.21.15.113:0/838835374 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f34ac152cb0 0x7f34ac1601d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f349c009840 tx=0x7f349c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ab14ddc921a0cec server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f9ce2625700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f9cdc0d3080 0x7f9cdc163580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f9ce2625700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9cdc0b6d20 0x7f9cdc0b7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f9ce2625700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cdc0b76b0 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.383+0000 7f9ce2625700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cdc0b3170 con 0x7f9cdc0b6d20 2024-01-15T11:25:22.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f9ce2625700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cdc0d3460 con 0x7f9cdc0d3080 2024-01-15T11:25:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f9cdb7fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f9cdc0b9bf0 0x7f9cdc0b2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:37530/0 (socket says 172.21.15.113:37530) 2024-01-15T11:25:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f9cdb7fe700 1 -- 172.21.15.113:0/420871939 learned_addr learned my addr 172.21.15.113:0/420871939 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f9ce0bc2700 1 --2- 172.21.15.113:0/420871939 >> v2:172.21.15.113:3300/0 conn(0x7f9cdc0b6d20 0x7f9cdc0b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f9cdbfff700 1 --2- 172.21.15.113:0/420871939 >> v2:172.21.15.141:3300/0 conn(0x7f9cdc0d3080 0x7f9cdc163580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f9ce0bc2700 1 -- 172.21.15.113:0/420871939 >> v2:172.21.15.141:3300/0 conn(0x7f9cdc0d3080 msgr2=0x7f9cdc163580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f9ce0bc2700 1 --2- 172.21.15.113:0/420871939 >> v2:172.21.15.141:3300/0 conn(0x7f9cdc0d3080 0x7f9cdc163580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f9ce0bc2700 1 -- 172.21.15.113:0/420871939 >> v1:172.21.15.113:6789/0 conn(0x7f9cdc0b9bf0 legacy=0x7f9cdc0b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f9ce0bc2700 1 -- 172.21.15.113:0/420871939 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc0b3390 con 0x7f9cdc0b6d20 2024-01-15T11:25:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.384+0000 7f9ce0bc2700 1 --2- 172.21.15.113:0/420871939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0b6d20 0x7f9cdc0b7170 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0009840 tx=0x7f9cd0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12786536729879da server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.385+0000 7f9cdaffd700 1 -- 172.21.15.113:0/420871939 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0011830 con 0x7f9cdc0b6d20 2024-01-15T11:25:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.386+0000 7f9cdaffd700 1 -- 172.21.15.113:0/420871939 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd0011990 con 0x7f9cdc0b6d20 2024-01-15T11:25:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.386+0000 7f9cdaffd700 1 -- 172.21.15.113:0/420871939 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0011d00 con 0x7f9cdc0b6d20 2024-01-15T11:25:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.386+0000 7f34ab7fe700 1 -- 172.21.15.113:0/838835374 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c011830 con 0x7f34ac152cb0 2024-01-15T11:25:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.386+0000 7f34ab7fe700 1 -- 172.21.15.113:0/838835374 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c011990 con 0x7f34ac152cb0 2024-01-15T11:25:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.386+0000 7f34ab7fe700 1 -- 172.21.15.113:0/838835374 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c011d00 con 0x7f34ac152cb0 2024-01-15T11:25:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.386+0000 7f6088ff9700 1 -- 172.21.15.113:0/1048633126 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6080003b90 con 0x7f608c0b3d40 2024-01-15T11:25:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.386+0000 7f6088ff9700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f607806b450 0x7f607806d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f6088ff9700 1 -- 172.21.15.113:0/1048633126 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60800895c0 con 0x7f608c0b3d40 2024-01-15T11:25:22.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f34b18b7700 1 -- 172.21.15.113:0/838835374 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f34ac152cb0 msgr2=0x7f34ac1601d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f34b18b7700 1 --2- 172.21.15.113:0/838835374 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f34ac152cb0 0x7f34ac1601d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f349c009840 tx=0x7f349c009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:22.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f9ce2625700 1 -- 172.21.15.113:0/420871939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0b6d20 msgr2=0x7f9cdc0b7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f9ce2625700 1 --2- 172.21.15.113:0/420871939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0b6d20 0x7f9cdc0b7170 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0009840 tx=0x7f9cd0009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:22.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7fe52f7fe700 1 -- 172.21.15.113:0/3329661122 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5380544a0 con 0x7fe530005460 2024-01-15T11:25:22.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f9ce2625700 1 -- 172.21.15.113:0/420871939 shutdown_connections 2024-01-15T11:25:22.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f9ce2625700 1 --2- 172.21.15.113:0/420871939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0b6d20 0x7f9cdc0b7170 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f9ce2625700 1 --2- 172.21.15.113:0/420871939 >> v2:172.21.15.141:3300/0 conn(0x7f9cdc0d3080 0x7f9cdc163580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f9ce2625700 1 -- 172.21.15.113:0/420871939 >> 172.21.15.113:0/420871939 conn(0x7f9cdc0d3ed0 msgr2=0x7f9cdc0d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f34b18b7700 1 -- 172.21.15.113:0/838835374 shutdown_connections 2024-01-15T11:25:22.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f34b18b7700 1 --2- 172.21.15.113:0/838835374 >> v2:172.21.15.141:3300/0 conn(0x7f34ac0ae6c0 0x7f34ac0aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f34b18b7700 1 --2- 172.21.15.113:0/838835374 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f34ac152cb0 0x7f34ac1601d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f34b18b7700 1 -- 172.21.15.113:0/838835374 >> 172.21.15.113:0/838835374 conn(0x7f34ac074780 msgr2=0x7f34ac06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f34b18b7700 1 -- 172.21.15.113:0/838835374 shutdown_connections 2024-01-15T11:25:22.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.387+0000 7f34b18b7700 1 -- 172.21.15.113:0/838835374 wait complete. 2024-01-15T11:25:22.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.388+0000 7f60b71a4700 1 Processor -- start 2024-01-15T11:25:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.388+0000 7f34b18b7700 1 Processor -- start 2024-01-15T11:25:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.388+0000 7f34b18b7700 1 -- start start 2024-01-15T11:25:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.389+0000 7f60b71a4700 1 -- start start 2024-01-15T11:25:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.389+0000 7f34b18b7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34ac0ae6c0 0x7f34ac138180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.389+0000 7f34b18b7700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f34ac152cb0 0x7f34ac071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.389+0000 7f34b18b7700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34ac15c1e0 0x7f34ac071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.389+0000 7f34b18b7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0ce7a0 con 0x7f34ac0ae6c0 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.389+0000 7f34b18b7700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0cdbb0 con 0x7f34ac15c1e0 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.389+0000 7f34b18b7700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0cde70 con 0x7f34ac152cb0 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f34b10b6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34ac15c1e0 0x7f34ac071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f34abfff700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f34ac152cb0 0x7f34ac071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f34b10b6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34ac15c1e0 0x7f34ac071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51450/0 (socket says 172.21.15.113:51450) 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f34b10b6700 1 -- 172.21.15.113:0/2757918503 learned_addr learned my addr 172.21.15.113:0/2757918503 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f34b08b5700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34ac0ae6c0 0x7f34ac138180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f34b10b6700 1 -- 172.21.15.113:0/2757918503 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f34ac152cb0 msgr2=0x7f34ac071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f34b10b6700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f34ac152cb0 0x7f34ac071960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f34b10b6700 1 -- 172.21.15.113:0/2757918503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34ac0ae6c0 msgr2=0x7f34ac138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f34b10b6700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34ac0ae6c0 0x7f34ac138180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f34b10b6700 1 -- 172.21.15.113:0/2757918503 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c0094a0 con 0x7f34ac15c1e0 2024-01-15T11:25:22.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f60b71a4700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f60b0158a50 0x7f60b00c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f60b71a4700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f60b0162a30 0x7f60b00c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f60b71a4700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b00cfff0 con 0x7f60b0158670 2024-01-15T11:25:22.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f60b71a4700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b00629b0 con 0x7f60b0162a30 2024-01-15T11:25:22.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.390+0000 7f60b71a4700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b00cfd10 con 0x7f60b0158a50 2024-01-15T11:25:22.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.391+0000 7f34b10b6700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34ac15c1e0 0x7f34ac071ea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f34a400efc0 tx=0x7f34a400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.391+0000 7f60b4f40700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f60b0158670 0x7f60b00622a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:37546/0 (socket says 172.21.15.113:37546) 2024-01-15T11:25:22.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.391+0000 7f60b4f40700 1 -- 172.21.15.113:0/1815357366 learned_addr learned my addr 172.21.15.113:0/1815357366 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:22.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.391+0000 7f60b5741700 1 --2- 172.21.15.113:0/1815357366 >> v2:172.21.15.113:3300/0 conn(0x7f60b0162a30 0x7f60b00c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.391+0000 7f60affff700 1 --2- 172.21.15.113:0/1815357366 >> v2:172.21.15.141:3300/0 conn(0x7f60b0158a50 0x7f60b00c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.391+0000 7f60b5741700 1 -- 172.21.15.113:0/1815357366 >> v2:172.21.15.141:3300/0 conn(0x7f60b0158a50 msgr2=0x7f60b00c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.391+0000 7f60b5741700 1 --2- 172.21.15.113:0/1815357366 >> v2:172.21.15.141:3300/0 conn(0x7f60b0158a50 0x7f60b00c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.391+0000 7f60b5741700 1 -- 172.21.15.113:0/1815357366 >> v1:172.21.15.113:6789/0 conn(0x7f60b0158670 legacy=0x7f60b00622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.391+0000 7f60b5741700 1 -- 172.21.15.113:0/1815357366 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b0161f50 con 0x7f60b0162a30 2024-01-15T11:25:22.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.392+0000 7f60b5741700 1 --2- 172.21.15.113:0/1815357366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60b0162a30 0x7f60b00c9df0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f60b00ca730 tx=0x7f60a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d318537ef7c588bb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.392+0000 7f9ce2625700 1 -- 172.21.15.113:0/420871939 shutdown_connections 2024-01-15T11:25:22.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.392+0000 7f34a97fa700 1 -- 172.21.15.113:0/2757918503 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a4009ea0 con 0x7f34ac15c1e0 2024-01-15T11:25:22.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.392+0000 7f34a97fa700 1 -- 172.21.15.113:0/2757918503 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a400f040 con 0x7f34ac15c1e0 2024-01-15T11:25:22.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.392+0000 7f34a97fa700 1 -- 172.21.15.113:0/2757918503 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a4014480 con 0x7f34ac15c1e0 2024-01-15T11:25:22.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.392+0000 7f34b18b7700 1 -- 172.21.15.113:0/2757918503 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ac0ce150 con 0x7f34ac15c1e0 2024-01-15T11:25:22.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.392+0000 7f34b18b7700 1 -- 172.21.15.113:0/2757918503 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ac1386c0 con 0x7f34ac15c1e0 2024-01-15T11:25:22.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.394+0000 7f34b18b7700 1 -- 172.21.15.113:0/2757918503 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ac0ce950 con 0x7f34ac15c1e0 2024-01-15T11:25:22.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f34a97fa700 1 -- 172.21.15.113:0/2757918503 <== mon.1 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f34a40145e0 con 0x7f34ac15c1e0 2024-01-15T11:25:22.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.392+0000 7f9ce2625700 1 -- 172.21.15.113:0/420871939 wait complete. 2024-01-15T11:25:22.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.394+0000 7f9ce2625700 1 Processor -- start 2024-01-15T11:25:22.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.394+0000 7f60af7fe700 1 -- 172.21.15.113:0/1815357366 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a800e070 con 0x7f60b0162a30 2024-01-15T11:25:22.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.394+0000 7f60af7fe700 1 -- 172.21.15.113:0/1815357366 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a800bbd0 con 0x7f60b0162a30 2024-01-15T11:25:22.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.394+0000 7f60af7fe700 1 -- 172.21.15.113:0/1815357366 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8012430 con 0x7f60b0162a30 2024-01-15T11:25:22.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f60b71a4700 1 -- 172.21.15.113:0/1815357366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60b0162a30 msgr2=0x7f60b00c9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f60b71a4700 1 --2- 172.21.15.113:0/1815357366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60b0162a30 0x7f60b00c9df0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f60b00ca730 tx=0x7f60a800b330 comp rx=0 tx=0).stop 2024-01-15T11:25:22.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f60b71a4700 1 -- 172.21.15.113:0/1815357366 shutdown_connections 2024-01-15T11:25:22.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f60b71a4700 1 --2- 172.21.15.113:0/1815357366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60b0162a30 0x7f60b00c9df0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f60b71a4700 1 --2- 172.21.15.113:0/1815357366 >> v2:172.21.15.141:3300/0 conn(0x7f60b0158a50 0x7f60b00c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f60b71a4700 1 -- 172.21.15.113:0/1815357366 >> 172.21.15.113:0/1815357366 conn(0x7f60b00b57b0 msgr2=0x7f60b00b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f34a97fa700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f348c06b420 0x7f348c06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f34a97fa700 1 -- 172.21.15.113:0/2757918503 <== mon.1 v2:172.21.15.141:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34a408a6a0 con 0x7f34ac15c1e0 2024-01-15T11:25:22.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f84b8bc2700 1 Processor -- start 2024-01-15T11:25:22.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f9ce2625700 1 -- start start 2024-01-15T11:25:22.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.394+0000 7f6088ff9700 1 -- 172.21.15.113:0/1048633126 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6080054380 con 0x7f608c0b3d40 2024-01-15T11:25:22.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f608bfff700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f607806b450 0x7f607806d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.395+0000 7f608bfff700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f607806b450 0x7f607806d910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f607c007fd0 tx=0x7f607c007ec0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.396+0000 7f34b08b5700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f348c06b420 0x7f348c06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f34b08b5700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f348c06b420 0x7f348c06d8e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f349c0090b0 tx=0x7f349c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.396+0000 7f9ce2625700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9cdc0b6d20 0x7f9cdc1fffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.396+0000 7f9ce2625700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9cdc0b9bf0 0x7f9cdc0c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.396+0000 7f9ce2625700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0d3080 0x7f9cdc160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.396+0000 7f9ce2625700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc070550 con 0x7f9cdc0d3080 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.396+0000 7f9ce2625700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc200520 con 0x7f9cdc0b6d20 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.396+0000 7f9ce2625700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc0c53c0 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.396+0000 7f9cdbfff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9cdc0b6d20 0x7f9cdc1fffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.396+0000 7f9cdbfff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9cdc0b6d20 0x7f9cdc1fffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51478/0 (socket says 172.21.15.113:51478) 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.396+0000 7f9cdbfff700 1 -- 172.21.15.113:0/3301351180 learned_addr learned my addr 172.21.15.113:0/3301351180 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f9cdb7fe700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9cdc0b9bf0 0x7f9cdc0c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f9ce0bc2700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0d3080 0x7f9cdc160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f9cdb7fe700 1 -- 172.21.15.113:0/3301351180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9cdc0b6d20 msgr2=0x7f9cdc1fffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f9cdb7fe700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9cdc0b6d20 0x7f9cdc1fffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f9cdb7fe700 1 -- 172.21.15.113:0/3301351180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0d3080 msgr2=0x7f9cdc160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f9cdb7fe700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0d3080 0x7f9cdc160a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f9cdb7fe700 1 -- 172.21.15.113:0/3301351180 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd00094a0 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.396+0000 7f84b8bc2700 1 -- start start 2024-01-15T11:25:22.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f84b8bc2700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f84b4072df0 0x7f84b40b1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f84b8bc2700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f84b40b0230 0x7f84b40cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f84b8bc2700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b40ccde0 con 0x7f84b415ddf0 2024-01-15T11:25:22.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f84b8bc2700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b40cd0c0 con 0x7f84b4072df0 2024-01-15T11:25:22.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f84b8bc2700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b40cd3a0 con 0x7f84b40b0230 2024-01-15T11:25:22.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f9cdb7fe700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9cdc0b9bf0 0x7f9cdc0c9b50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400ca70 tx=0x7f9cd400cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f60b71a4700 1 -- 172.21.15.113:0/1815357366 shutdown_connections 2024-01-15T11:25:22.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f84b259c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f84b415ddf0 0x7f84b40726e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:37550/0 (socket says 172.21.15.113:37550) 2024-01-15T11:25:22.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f84b259c700 1 -- 172.21.15.113:0/2752891543 learned_addr learned my addr 172.21.15.113:0/2752891543 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:22.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.398+0000 7f9cd8ff9700 1 -- 172.21.15.113:0/3301351180 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd400a980 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.398+0000 7f9cd8ff9700 1 -- 172.21.15.113:0/3301351180 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd400aae0 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.398+0000 7f9cd8ff9700 1 -- 172.21.15.113:0/3301351180 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd400ae50 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.399+0000 7f84b2d9d700 1 --2- 172.21.15.113:0/2752891543 >> v2:172.21.15.141:3300/0 conn(0x7f84b40b0230 0x7f84b40cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.397+0000 7f60b71a4700 1 -- 172.21.15.113:0/1815357366 wait complete. 2024-01-15T11:25:22.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.399+0000 7f60b71a4700 1 Processor -- start 2024-01-15T11:25:22.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.399+0000 7f9ce2625700 1 -- 172.21.15.113:0/3301351180 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cdc0c56a0 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.399+0000 7f9ce2625700 1 -- 172.21.15.113:0/3301351180 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cdc0c5c60 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f84b1d9b700 1 --2- 172.21.15.113:0/2752891543 >> v2:172.21.15.113:3300/0 conn(0x7f84b4072df0 0x7f84b40b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.399+0000 7f60b71a4700 1 -- start start 2024-01-15T11:25:22.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f60b71a4700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f60b0158670 0x7f60b00adc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f60b71a4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60b0158a50 0x7f60b00b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f60b71a4700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0162a30 0x7f60b0157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f60b71a4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b01fc6e0 con 0x7f60b0158a50 2024-01-15T11:25:22.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f60b71a4700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b01fb7f0 con 0x7f60b0162a30 2024-01-15T11:25:22.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f60b71a4700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b01fbad0 con 0x7f60b0158670 2024-01-15T11:25:22.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f84b1d9b700 1 -- 172.21.15.113:0/2752891543 >> v2:172.21.15.141:3300/0 conn(0x7f84b40b0230 msgr2=0x7f84b40cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f84b1d9b700 1 --2- 172.21.15.113:0/2752891543 >> v2:172.21.15.141:3300/0 conn(0x7f84b40b0230 0x7f84b40cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f84b1d9b700 1 -- 172.21.15.113:0/2752891543 >> v1:172.21.15.113:6789/0 conn(0x7f84b415ddf0 legacy=0x7f84b40726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f84b1d9b700 1 -- 172.21.15.113:0/2752891543 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b40cd680 con 0x7f84b4072df0 2024-01-15T11:25:22.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f60affff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60b0158a50 0x7f60b00b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f60affff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60b0158a50 0x7f60b00b4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45732/0 (socket says 172.21.15.113:45732) 2024-01-15T11:25:22.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f60affff700 1 -- 172.21.15.113:0/933031356 learned_addr learned my addr 172.21.15.113:0/933031356 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.401+0000 7f84b1d9b700 1 --2- 172.21.15.113:0/2752891543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84b4072df0 0x7f84b40b1320 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f84b4074a90 tx=0x7f84ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7b76f667913ac2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.402+0000 7f9ce2625700 1 -- 172.21.15.113:0/3301351180 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cdc067f60 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.402+0000 7f60affff700 1 -- 172.21.15.113:0/933031356 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f60b0158670 msgr2=0x7f60b00adc80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T11:25:22.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.402+0000 7f60affff700 1 --2- 172.21.15.113:0/933031356 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f60b0158670 0x7f60b00adc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.402+0000 7f60affff700 1 -- 172.21.15.113:0/933031356 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0162a30 msgr2=0x7f60b0157410 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T11:25:22.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.402+0000 7f60affff700 1 --2- 172.21.15.113:0/933031356 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0162a30 0x7f60b0157410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.402+0000 7f60affff700 1 -- 172.21.15.113:0/933031356 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a8009a70 con 0x7f60b0158a50 2024-01-15T11:25:22.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.402+0000 7f60affff700 1 --2- 172.21.15.113:0/933031356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60b0158a50 0x7f60b00b4a60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f609c00e810 tx=0x7f609c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.402+0000 7f60ad7fa700 1 -- 172.21.15.113:0/933031356 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c013610 con 0x7f60b0158a50 2024-01-15T11:25:22.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.402+0000 7f60ad7fa700 1 -- 172.21.15.113:0/933031356 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c013770 con 0x7f60b0158a50 2024-01-15T11:25:22.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.402+0000 7f60ad7fa700 1 -- 172.21.15.113:0/933031356 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c013b20 con 0x7f60b0158a50 2024-01-15T11:25:22.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.402+0000 7f60b71a4700 1 -- 172.21.15.113:0/933031356 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b01fbdb0 con 0x7f60b0158a50 2024-01-15T11:25:22.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.403+0000 7f60b71a4700 1 -- 172.21.15.113:0/933031356 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b01fc370 con 0x7f60b0158a50 2024-01-15T11:25:22.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.404+0000 7f34a97fa700 1 -- 172.21.15.113:0/2757918503 <== mon.1 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34ac0ce950 con 0x7f34ac15c1e0 2024-01-15T11:25:22.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.407+0000 7f9cd8ff9700 1 -- 172.21.15.113:0/3301351180 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9cd401b3b0 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.407+0000 7f9cd8ff9700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f9cc806b420 0x7f9cc806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.407+0000 7f9cd8ff9700 1 -- 172.21.15.113:0/3301351180 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cd4089300 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.408+0000 7f9cdbfff700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f9cc806b420 0x7f9cc806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.408+0000 7f9cdbfff700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f9cc806b420 0x7f9cc806d8e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00ac10 tx=0x7f9ccc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.405+0000 7f60b71a4700 1 -- 172.21.15.113:0/933031356 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b0067f60 con 0x7f60b0158a50 2024-01-15T11:25:22.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.409+0000 7f84b159a700 1 -- 172.21.15.113:0/2752891543 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac00bcd0 con 0x7f84b4072df0 2024-01-15T11:25:22.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.409+0000 7f84b159a700 1 -- 172.21.15.113:0/2752891543 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac00be30 con 0x7f84b4072df0 2024-01-15T11:25:22.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.409+0000 7f84b159a700 1 -- 172.21.15.113:0/2752891543 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac003c60 con 0x7f84b4072df0 2024-01-15T11:25:22.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.410+0000 7f84b8bc2700 1 -- 172.21.15.113:0/2752891543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84b4072df0 msgr2=0x7f84b40b1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.410+0000 7f84b8bc2700 1 --2- 172.21.15.113:0/2752891543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84b4072df0 0x7f84b40b1320 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f84b4074a90 tx=0x7f84ac00b330 comp rx=0 tx=0).stop 2024-01-15T11:25:22.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.409+0000 7f60ad7fa700 1 -- 172.21.15.113:0/933031356 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f609c013c80 con 0x7f60b0158a50 2024-01-15T11:25:22.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.409+0000 7f60ad7fa700 1 --2- 172.21.15.113:0/933031356 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f609406b420 0x7f609406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.409+0000 7f60ad7fa700 1 -- 172.21.15.113:0/933031356 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f609c089800 con 0x7f60b0158a50 2024-01-15T11:25:22.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.410+0000 7f60b4f40700 1 --2- 172.21.15.113:0/933031356 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f609406b420 0x7f609406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.410+0000 7f84b8bc2700 1 -- 172.21.15.113:0/2752891543 shutdown_connections 2024-01-15T11:25:22.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.410+0000 7f84b8bc2700 1 --2- 172.21.15.113:0/2752891543 >> v2:172.21.15.141:3300/0 conn(0x7f84b40b0230 0x7f84b40cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.410+0000 7f84b8bc2700 1 --2- 172.21.15.113:0/2752891543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84b4072df0 0x7f84b40b1320 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.410+0000 7f84b8bc2700 1 -- 172.21.15.113:0/2752891543 >> 172.21.15.113:0/2752891543 conn(0x7f84b40b57b0 msgr2=0x7f84b40b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.410+0000 7f60b4f40700 1 --2- 172.21.15.113:0/933031356 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f609406b420 0x7f609406d8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f60a4001600 tx=0x7f60a4007ad0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.410+0000 7f84b8bc2700 1 -- 172.21.15.113:0/2752891543 shutdown_connections 2024-01-15T11:25:22.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.411+0000 7f84b8bc2700 1 -- 172.21.15.113:0/2752891543 wait complete. 2024-01-15T11:25:22.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.413+0000 7f9cd8ff9700 1 -- 172.21.15.113:0/3301351180 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9cd4054290 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.413+0000 7f84b8bc2700 1 Processor -- start 2024-01-15T11:25:22.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.413+0000 7f84b8bc2700 1 -- start start 2024-01-15T11:25:22.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.414+0000 7f84b8bc2700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f84b4072df0 0x7f84b40aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.414+0000 7f84b8bc2700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84b40b0230 0x7f84b41e2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.414+0000 7f84b8bc2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84b415ddf0 0x7f84b41f8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.414+0000 7f84b8bc2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b415c8c0 con 0x7f84b415ddf0 2024-01-15T11:25:22.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.414+0000 7f84b8bc2700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b41e3420 con 0x7f84b40b0230 2024-01-15T11:25:22.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.414+0000 7f84b8bc2700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b41f8660 con 0x7f84b4072df0 2024-01-15T11:25:22.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.414+0000 7f84b1d9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84b40b0230 0x7f84b41e2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.414+0000 7f84b1d9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84b40b0230 0x7f84b41e2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51512/0 (socket says 172.21.15.113:51512) 2024-01-15T11:25:22.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.415+0000 7f84b1d9b700 1 -- 172.21.15.113:0/4257208812 learned_addr learned my addr 172.21.15.113:0/4257208812 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.415+0000 7f84b259c700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f84b4072df0 0x7f84b40aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.421+0000 7fe115043700 1 Processor -- start 2024-01-15T11:25:22.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.421+0000 7fe115043700 1 -- start start 2024-01-15T11:25:22.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.422+0000 7fe115043700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe110163480 0x7fe1100b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.422+0000 7fe115043700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe110163860 0x7fe1100b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.422+0000 7fe115043700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1100b2e80 con 0x7fe1100b1e40 2024-01-15T11:25:22.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.422+0000 7fe115043700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1100b2980 con 0x7fe110163480 2024-01-15T11:25:22.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.422+0000 7fe115043700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe110164510 con 0x7fe110163860 2024-01-15T11:25:22.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.422+0000 7fe10ed9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe110163480 0x7fe1100b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.422+0000 7fe10ed9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe110163480 0x7fe1100b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45748/0 (socket says 172.21.15.113:45748) 2024-01-15T11:25:22.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.423+0000 7fe10ed9d700 1 -- 172.21.15.113:0/1459620224 learned_addr learned my addr 172.21.15.113:0/1459620224 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.423+0000 7fe10e59c700 1 --2- 172.21.15.113:0/1459620224 >> v2:172.21.15.141:3300/0 conn(0x7fe110163860 0x7fe1100b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.423+0000 7fe10ed9d700 1 -- 172.21.15.113:0/1459620224 >> v2:172.21.15.141:3300/0 conn(0x7fe110163860 msgr2=0x7fe1100b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.423+0000 7fe10ed9d700 1 --2- 172.21.15.113:0/1459620224 >> v2:172.21.15.141:3300/0 conn(0x7fe110163860 0x7fe1100b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.423+0000 7fe10ed9d700 1 -- 172.21.15.113:0/1459620224 >> v1:172.21.15.113:6789/0 conn(0x7fe1100b1e40 legacy=0x7fe1100b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.423+0000 7fe10ed9d700 1 -- 172.21.15.113:0/1459620224 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe110160610 con 0x7fe110163480 2024-01-15T11:25:22.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.423+0000 7fe10e59c700 1 --2- 172.21.15.113:0/1459620224 >> v2:172.21.15.141:3300/0 conn(0x7fe110163860 0x7fe1100b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:22.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.423+0000 7fe10ed9d700 1 --2- 172.21.15.113:0/1459620224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110163480 0x7fe1100b01a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe11006fc10 tx=0x7fe0f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23fe4d9d893b754 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.424+0000 7fe10dd9b700 1 -- 172.21.15.113:0/1459620224 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f8004230 con 0x7fe110163480 2024-01-15T11:25:22.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.424+0000 7fe10dd9b700 1 -- 172.21.15.113:0/1459620224 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f8004390 con 0x7fe110163480 2024-01-15T11:25:22.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.424+0000 7fe10dd9b700 1 -- 172.21.15.113:0/1459620224 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f80036a0 con 0x7fe110163480 2024-01-15T11:25:22.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.425+0000 7fe115043700 1 -- 172.21.15.113:0/1459620224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110163480 msgr2=0x7fe1100b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.425+0000 7fe115043700 1 --2- 172.21.15.113:0/1459620224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110163480 0x7fe1100b01a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe11006fc10 tx=0x7fe0f8009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.424+0000 7f84b2d9d700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84b415ddf0 0x7f84b41f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.425+0000 7f84b259c700 1 -- 172.21.15.113:0/4257208812 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84b40b0230 msgr2=0x7f84b41e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.425+0000 7f84b259c700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84b40b0230 0x7f84b41e2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.425+0000 7f84b259c700 1 -- 172.21.15.113:0/4257208812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84b415ddf0 msgr2=0x7f84b41f8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.425+0000 7f84b259c700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84b415ddf0 0x7f84b41f8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.425+0000 7f84b259c700 1 -- 172.21.15.113:0/4257208812 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ac009a70 con 0x7f84b4072df0 2024-01-15T11:25:22.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.426+0000 7f84b259c700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f84b4072df0 0x7f84b40aeeb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f84a800c550 tx=0x7f84a800c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.426+0000 7fe115043700 1 -- 172.21.15.113:0/1459620224 shutdown_connections 2024-01-15T11:25:22.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.426+0000 7fe115043700 1 --2- 172.21.15.113:0/1459620224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110163480 0x7fe1100b01a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.426+0000 7fe115043700 1 --2- 172.21.15.113:0/1459620224 >> v2:172.21.15.141:3300/0 conn(0x7fe110163860 0x7fe1100b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.426+0000 7fe115043700 1 -- 172.21.15.113:0/1459620224 >> 172.21.15.113:0/1459620224 conn(0x7fe1100b57b0 msgr2=0x7fe1100b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.426+0000 7f84a2ffd700 1 -- 172.21.15.113:0/4257208812 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8004c80 con 0x7f84b4072df0 2024-01-15T11:25:22.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.426+0000 7f21196e7700 1 Processor -- start 2024-01-15T11:25:22.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.426+0000 7f84a2ffd700 1 -- 172.21.15.113:0/4257208812 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a8004e20 con 0x7f84b4072df0 2024-01-15T11:25:22.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.426+0000 7f84a2ffd700 1 -- 172.21.15.113:0/4257208812 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8007310 con 0x7f84b4072df0 2024-01-15T11:25:22.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.427+0000 7f60ad7fa700 1 -- 172.21.15.113:0/933031356 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f609c0546e0 con 0x7f60b0158a50 2024-01-15T11:25:22.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.426+0000 7f21196e7700 1 -- start start 2024-01-15T11:25:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.427+0000 7f21196e7700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2114163480 0x7f21140b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.427+0000 7f21196e7700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2114163860 0x7f21140b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.427+0000 7f21196e7700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21140b2e80 con 0x7f21140b1e40 2024-01-15T11:25:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.427+0000 7f21196e7700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21140b29c0 con 0x7f2114163480 2024-01-15T11:25:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.427+0000 7f21196e7700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2114164510 con 0x7f2114163860 2024-01-15T11:25:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.428+0000 7f2112ffd700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2114163480 0x7f21140b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.428+0000 7f2112ffd700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2114163480 0x7f21140b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51524/0 (socket says 172.21.15.113:51524) 2024-01-15T11:25:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.428+0000 7f2112ffd700 1 -- 172.21.15.113:0/1419042284 learned_addr learned my addr 172.21.15.113:0/1419042284 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.428+0000 7f21127fc700 1 --2- 172.21.15.113:0/1419042284 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2114163860 0x7f21140b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.428+0000 7f2112ffd700 1 -- 172.21.15.113:0/1419042284 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2114163860 msgr2=0x7f21140b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.428+0000 7f2112ffd700 1 --2- 172.21.15.113:0/1419042284 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2114163860 0x7f21140b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.428+0000 7f2112ffd700 1 -- 172.21.15.113:0/1419042284 >> v1:172.21.15.141:6789/0 conn(0x7f21140b1e40 legacy=0x7f21140b22b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T11:25:22.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.428+0000 7f2112ffd700 1 -- 172.21.15.113:0/1419042284 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2114160610 con 0x7f2114163480 2024-01-15T11:25:22.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.428+0000 7f21127fc700 1 --2- 172.21.15.113:0/1419042284 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2114163860 0x7f21140b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:22.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.428+0000 7f2112ffd700 1 --2- 172.21.15.113:0/1419042284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2114163480 0x7f21140b01a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f21080098f0 tx=0x7f2108009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77b5e434ae5d1b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.432+0000 7f2111ffb700 1 -- 172.21.15.113:0/1419042284 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2108004230 con 0x7f2114163480 2024-01-15T11:25:22.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.432+0000 7f2111ffb700 1 -- 172.21.15.113:0/1419042284 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2108004390 con 0x7f2114163480 2024-01-15T11:25:22.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.432+0000 7f2111ffb700 1 -- 172.21.15.113:0/1419042284 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21080036a0 con 0x7f2114163480 2024-01-15T11:25:22.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.433+0000 7f84b8bc2700 1 -- 172.21.15.113:0/4257208812 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b41e3650 con 0x7f84b4072df0 2024-01-15T11:25:22.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.433+0000 7f84b8bc2700 1 -- 172.21.15.113:0/4257208812 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b415bf20 con 0x7f84b4072df0 2024-01-15T11:25:22.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.433+0000 7fe115043700 1 -- 172.21.15.113:0/1459620224 shutdown_connections 2024-01-15T11:25:22.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.433+0000 7fe115043700 1 -- 172.21.15.113:0/1459620224 wait complete. 2024-01-15T11:25:22.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.434+0000 7f84b8bc2700 1 -- 172.21.15.113:0/4257208812 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b4067f60 con 0x7f84b4072df0 2024-01-15T11:25:22.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.435+0000 7fe115043700 1 Processor -- start 2024-01-15T11:25:22.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.435+0000 7fe115043700 1 -- start start 2024-01-15T11:25:22.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.435+0000 7fe115043700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe1100b1e40 0x7fe11014d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.435+0000 7fe115043700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe110163480 0x7fe110146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.435+0000 7fe115043700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110163860 0x7fe11014fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.435+0000 7fe115043700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1101506b0 con 0x7fe110163860 2024-01-15T11:25:22.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.435+0000 7fe115043700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe110150990 con 0x7fe110163480 2024-01-15T11:25:22.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.435+0000 7fe115043700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1100cada0 con 0x7fe1100b1e40 2024-01-15T11:25:22.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.436+0000 7fe10ed9d700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe1100b1e40 0x7fe11014d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.436+0000 7fe10e59c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe110163480 0x7fe110146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.436+0000 7fe10ed9d700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe1100b1e40 0x7fe11014d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:47436/0 (socket says 172.21.15.113:47436) 2024-01-15T11:25:22.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.436+0000 7fe10e59c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe110163480 0x7fe110146470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51532/0 (socket says 172.21.15.113:51532) 2024-01-15T11:25:22.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.436+0000 7fe10ed9d700 1 -- 172.21.15.113:0/2944394662 learned_addr learned my addr 172.21.15.113:0/2944394662 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.436+0000 7fe10e59c700 1 -- 172.21.15.113:0/2944394662 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe1100b1e40 msgr2=0x7fe11014d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.436+0000 7fe10e59c700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe1100b1e40 0x7fe11014d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.436+0000 7fe10e59c700 1 -- 172.21.15.113:0/2944394662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110163860 msgr2=0x7fe11014fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.436+0000 7fe10e59c700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110163860 0x7fe11014fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.436+0000 7fe10e59c700 1 -- 172.21.15.113:0/2944394662 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1000094a0 con 0x7fe110163480 2024-01-15T11:25:22.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.436+0000 7fe10ed9d700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe1100b1e40 0x7fe11014d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:22.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.436+0000 7fe10e59c700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe110163480 0x7fe110146470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe10000e900 tx=0x7fe10000ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.439+0000 7f21196e7700 1 -- 172.21.15.113:0/1419042284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2114163480 msgr2=0x7f21140b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.439+0000 7f21196e7700 1 --2- 172.21.15.113:0/1419042284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2114163480 0x7f21140b01a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f21080098f0 tx=0x7f2108009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:22.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.440+0000 7fe1077fe700 1 -- 172.21.15.113:0/2944394662 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10000ee50 con 0x7fe110163480 2024-01-15T11:25:22.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.440+0000 7fe1077fe700 1 -- 172.21.15.113:0/2944394662 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10000c850 con 0x7fe110163480 2024-01-15T11:25:22.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.440+0000 7fe1077fe700 1 -- 172.21.15.113:0/2944394662 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10000cc00 con 0x7fe110163480 2024-01-15T11:25:22.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.440+0000 7fe115043700 1 -- 172.21.15.113:0/2944394662 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f8009550 con 0x7fe110163480 2024-01-15T11:25:22.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.440+0000 7fe115043700 1 -- 172.21.15.113:0/2944394662 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11014dd50 con 0x7fe110163480 2024-01-15T11:25:22.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.440+0000 7f84a2ffd700 1 -- 172.21.15.113:0/4257208812 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f84a8007470 con 0x7f84b4072df0 2024-01-15T11:25:22.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.441+0000 7f84a2ffd700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f849406b350 0x7f849406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.441+0000 7f84a2ffd700 1 -- 172.21.15.113:0/4257208812 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84a8089df0 con 0x7f84b4072df0 2024-01-15T11:25:22.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.441+0000 7f21196e7700 1 -- 172.21.15.113:0/1419042284 shutdown_connections 2024-01-15T11:25:22.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.441+0000 7f21196e7700 1 --2- 172.21.15.113:0/1419042284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2114163480 0x7f21140b01a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.441+0000 7f21196e7700 1 --2- 172.21.15.113:0/1419042284 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2114163860 0x7f21140b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.441+0000 7f21196e7700 1 -- 172.21.15.113:0/1419042284 >> 172.21.15.113:0/1419042284 conn(0x7f21140b57b0 msgr2=0x7f21140b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.442+0000 7f21196e7700 1 -- 172.21.15.113:0/1419042284 shutdown_connections 2024-01-15T11:25:22.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.441+0000 7fe115043700 1 -- 172.21.15.113:0/2944394662 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f40052f0 con 0x7fe110163480 2024-01-15T11:25:22.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.442+0000 7fe1077fe700 1 -- 172.21.15.113:0/2944394662 <== mon.1 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe100013470 con 0x7fe110163480 2024-01-15T11:25:22.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.442+0000 7fe1077fe700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe0fc06b5b0 0x7fe0fc06da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.442+0000 7fe1077fe700 1 -- 172.21.15.113:0/2944394662 <== mon.1 v2:172.21.15.141:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe100089b90 con 0x7fe110163480 2024-01-15T11:25:22.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.442+0000 7f21196e7700 1 -- 172.21.15.113:0/1419042284 wait complete. 2024-01-15T11:25:22.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.444+0000 7f21196e7700 1 Processor -- start 2024-01-15T11:25:22.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.444+0000 7f84b1d9b700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f849406b350 0x7f849406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.444+0000 7f84a2ffd700 1 -- 172.21.15.113:0/4257208812 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84a8054d00 con 0x7f84b4072df0 2024-01-15T11:25:22.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.446+0000 7fe10ed9d700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe0fc06b5b0 0x7fe0fc06da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.445+0000 7f21196e7700 1 -- start start 2024-01-15T11:25:22.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.446+0000 7f21196e7700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f21140b1e40 0x7f21141669b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.446+0000 7f21196e7700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2114163480 0x7f2114155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.446+0000 7f21196e7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2114163860 0x7f21141f3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.446+0000 7f21196e7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2114164bf0 con 0x7f2114163860 2024-01-15T11:25:22.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.446+0000 7f21196e7700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21140cb000 con 0x7f2114163480 2024-01-15T11:25:22.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.446+0000 7f21196e7700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21140cb2e0 con 0x7f21140b1e40 2024-01-15T11:25:22.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.446+0000 7f2112ffd700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f21140b1e40 0x7f21141669b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.446+0000 7f21127fc700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2114163480 0x7f2114155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.446+0000 7f21127fc700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2114163480 0x7f2114155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51546/0 (socket says 172.21.15.113:51546) 2024-01-15T11:25:22.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.446+0000 7f21127fc700 1 -- 172.21.15.113:0/3504094886 learned_addr learned my addr 172.21.15.113:0/3504094886 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:22.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.447+0000 7f21137fe700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2114163860 0x7f21141f3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.447+0000 7fe1077fe700 1 -- 172.21.15.113:0/2944394662 <== mon.1 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1000549e0 con 0x7fe110163480 2024-01-15T11:25:22.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.447+0000 7f2112ffd700 1 -- 172.21.15.113:0/3504094886 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2114163480 msgr2=0x7f2114155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.447+0000 7f2112ffd700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2114163480 0x7f2114155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.447+0000 7f2112ffd700 1 -- 172.21.15.113:0/3504094886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2114163860 msgr2=0x7f21141f3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.447+0000 7f2112ffd700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2114163860 0x7f21141f3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.447+0000 7f2112ffd700 1 -- 172.21.15.113:0/3504094886 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21040094a0 con 0x7f21140b1e40 2024-01-15T11:25:22.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.448+0000 7f84b1d9b700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f849406b350 0x7f849406d810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f84b40d0cc0 tx=0x7f84ac013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.448+0000 7f2112ffd700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f21140b1e40 0x7f21141669b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2108015000 tx=0x7f2108004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.452+0000 7fe10ed9d700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe0fc06b5b0 0x7fe0fc06da70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009c10 tx=0x7fe0f8010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.458+0000 7f20fb7fe700 1 -- 172.21.15.113:0/3504094886 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2108003c30 con 0x7f21140b1e40 2024-01-15T11:25:22.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.458+0000 7f21196e7700 1 -- 172.21.15.113:0/3504094886 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2108009550 con 0x7f21140b1e40 2024-01-15T11:25:22.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.458+0000 7f21196e7700 1 -- 172.21.15.113:0/3504094886 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21140cb8a0 con 0x7f21140b1e40 2024-01-15T11:25:22.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.458+0000 7f20fb7fe700 1 -- 172.21.15.113:0/3504094886 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2108003db0 con 0x7f21140b1e40 2024-01-15T11:25:22.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.458+0000 7f20fb7fe700 1 -- 172.21.15.113:0/3504094886 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2108011870 con 0x7f21140b1e40 2024-01-15T11:25:22.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.464+0000 7f20fb7fe700 1 -- 172.21.15.113:0/3504094886 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2108022070 con 0x7f21140b1e40 2024-01-15T11:25:22.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.464+0000 7f20fb7fe700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f210006b350 0x7f210006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:22.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.465+0000 7f20fb7fe700 1 -- 172.21.15.113:0/3504094886 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f210808b480 con 0x7f21140b1e40 2024-01-15T11:25:22.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.465+0000 7f21127fc700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f210006b350 0x7f210006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:22.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.466+0000 7f21127fc700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f210006b350 0x7f210006d810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f21141564b0 tx=0x7f2104009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:22.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.466+0000 7f21196e7700 1 -- 172.21.15.113:0/3504094886 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f40052f0 con 0x7f21140b1e40 2024-01-15T11:25:22.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.471+0000 7f20fb7fe700 1 -- 172.21.15.113:0/3504094886 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21080562e0 con 0x7f21140b1e40 2024-01-15T11:25:22.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.792+0000 7fe547fff700 1 -- 172.21.15.113:0/3329661122 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe5480751e0 con 0x7fe530005460 2024-01-15T11:25:22.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.792+0000 7f34b18b7700 1 -- 172.21.15.113:0/2757918503 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f34ac138940 con 0x7f34ac15c1e0 2024-01-15T11:25:22.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.792+0000 7fe52f7fe700 1 -- 172.21.15.113:0/3329661122 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe538057ac0 con 0x7fe530005460 2024-01-15T11:25:22.791 INFO:teuthology.orchestra.run.smithi113.stdout:55834574854 2024-01-15T11:25:22.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7f34a97fa700 1 -- 172.21.15.113:0/2757918503 <== mon.1 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f34a4055580 con 0x7f34ac15c1e0 2024-01-15T11:25:22.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7fe547fff700 1 -- 172.21.15.113:0/3329661122 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe53406b420 msgr2=0x7fe53406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7fe547fff700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe53406b420 0x7fe53406d8e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00ba70 tx=0x7fe53c008040 comp rx=0 tx=0).stop 2024-01-15T11:25:22.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7fe547fff700 1 -- 172.21.15.113:0/3329661122 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 msgr2=0x7fe54808aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7fe547fff700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 0x7fe54808aba0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe53800caf0 tx=0x7fe53800ce00 comp rx=0 tx=0).stop 2024-01-15T11:25:22.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7fe547fff700 1 -- 172.21.15.113:0/3329661122 shutdown_connections 2024-01-15T11:25:22.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7fe547fff700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe53406b420 0x7fe53406d8e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7fe547fff700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe548142bb0 0x7fe548089a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7fe547fff700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe530005460 0x7fe54808aba0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7fe547fff700 1 --2- 172.21.15.113:0/3329661122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5480b5e20 0x7fe548142670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7fe547fff700 1 -- 172.21.15.113:0/3329661122 >> 172.21.15.113:0/3329661122 conn(0x7fe5480cd340 msgr2=0x7fe54815bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7fe547fff700 1 -- 172.21.15.113:0/3329661122 shutdown_connections 2024-01-15T11:25:22.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.796+0000 7fe547fff700 1 -- 172.21.15.113:0/3329661122 wait complete. 2024-01-15T11:25:22.798 INFO:teuthology.orchestra.run.smithi113.stdout:55834574855 2024-01-15T11:25:22.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.803+0000 7f9ce2625700 1 -- 172.21.15.113:0/3301351180 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9cdc0b76b0 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.803+0000 7f34b18b7700 1 -- 172.21.15.113:0/2757918503 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f348c06b420 msgr2=0x7f348c06d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.803+0000 7f34b18b7700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f348c06b420 0x7f348c06d8e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f349c0090b0 tx=0x7f349c008030 comp rx=0 tx=0).stop 2024-01-15T11:25:22.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.803+0000 7f34b18b7700 1 -- 172.21.15.113:0/2757918503 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34ac15c1e0 msgr2=0x7f34ac071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.803+0000 7f34b18b7700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34ac15c1e0 0x7f34ac071ea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f34a400efc0 tx=0x7f34a400c3a0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.803+0000 7f34b18b7700 1 -- 172.21.15.113:0/2757918503 shutdown_connections 2024-01-15T11:25:22.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.803+0000 7f34b18b7700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f348c06b420 0x7f348c06d8e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.803+0000 7f34b18b7700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34ac15c1e0 0x7f34ac071ea0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.803+0000 7f34b18b7700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34ac0ae6c0 0x7f34ac138180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.803+0000 7f34b18b7700 1 --2- 172.21.15.113:0/2757918503 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f34ac152cb0 0x7f34ac071960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.803+0000 7f34b18b7700 1 -- 172.21.15.113:0/2757918503 >> 172.21.15.113:0/2757918503 conn(0x7f34ac074780 msgr2=0x7f34ac0b0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.803+0000 7f9cd8ff9700 1 -- 172.21.15.113:0/3301351180 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9cdc0b76b0 con 0x7f9cdc0b9bf0 2024-01-15T11:25:22.803 INFO:teuthology.orchestra.run.smithi113.stdout:55834574854 2024-01-15T11:25:22.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.805+0000 7f34b18b7700 1 -- 172.21.15.113:0/2757918503 shutdown_connections 2024-01-15T11:25:22.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.805+0000 7f34b18b7700 1 -- 172.21.15.113:0/2757918503 wait complete. 2024-01-15T11:25:22.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.809+0000 7f9ce2625700 1 -- 172.21.15.113:0/3301351180 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f9cc806b420 msgr2=0x7f9cc806d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.809+0000 7f9ce2625700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f9cc806b420 0x7f9cc806d8e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00ac10 tx=0x7f9ccc008040 comp rx=0 tx=0).stop 2024-01-15T11:25:22.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.809+0000 7f9ce2625700 1 -- 172.21.15.113:0/3301351180 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9cdc0b9bf0 msgr2=0x7f9cdc0c9b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.810+0000 7f9ce2625700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9cdc0b9bf0 0x7f9cdc0c9b50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400ca70 tx=0x7f9cd400cd80 comp rx=0 tx=0).stop 2024-01-15T11:25:22.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.809+0000 7f6092776700 1 -- 172.21.15.113:0/1048633126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f606c005870 con 0x7f608c0b3d40 2024-01-15T11:25:22.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.819+0000 7f6088ff9700 1 -- 172.21.15.113:0/1048633126 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6080017090 con 0x7f608c0b3d40 2024-01-15T11:25:22.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.819+0000 7f9ce2625700 1 -- 172.21.15.113:0/3301351180 shutdown_connections 2024-01-15T11:25:22.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.819+0000 7f9ce2625700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f9cc806b420 0x7f9cc806d8e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.819+0000 7f9ce2625700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0d3080 0x7f9cdc160a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.819+0000 7f9ce2625700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9cdc0b6d20 0x7f9cdc1fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.819+0000 7f9ce2625700 1 --2- 172.21.15.113:0/3301351180 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9cdc0b9bf0 0x7f9cdc0c9b50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.819+0000 7f9ce2625700 1 -- 172.21.15.113:0/3301351180 >> 172.21.15.113:0/3301351180 conn(0x7f9cdc0d3ed0 msgr2=0x7f9cdc0b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.820 INFO:teuthology.orchestra.run.smithi113.stdout:55834574855 2024-01-15T11:25:22.821 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.2 2024-01-15T11:25:22.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.821+0000 7f9ce2625700 1 -- 172.21.15.113:0/3301351180 shutdown_connections 2024-01-15T11:25:22.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.822+0000 7f9ce2625700 1 -- 172.21.15.113:0/3301351180 wait complete. 2024-01-15T11:25:22.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.823+0000 7f6092776700 1 -- 172.21.15.113:0/1048633126 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f607806b450 msgr2=0x7f607806d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.823+0000 7f6092776700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f607806b450 0x7f607806d910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f607c007fd0 tx=0x7f607c007ec0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.823+0000 7f6092776700 1 -- 172.21.15.113:0/1048633126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f608c0b3d40 msgr2=0x7f608c0c49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.823+0000 7f6092776700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f608c0b3d40 0x7f608c0c49b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f608000c680 tx=0x7f608000c990 comp rx=0 tx=0).stop 2024-01-15T11:25:22.823 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.6 2024-01-15T11:25:22.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.825+0000 7f6092776700 1 -- 172.21.15.113:0/1048633126 shutdown_connections 2024-01-15T11:25:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.825+0000 7f6092776700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f607806b450 0x7f607806d910 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.825+0000 7f6092776700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f608c140680 0x7f608c0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.825+0000 7f6092776700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f608c071470 0x7f608c14ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.825+0000 7f6092776700 1 --2- 172.21.15.113:0/1048633126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f608c0b3d40 0x7f608c0c49b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.825+0000 7f6092776700 1 -- 172.21.15.113:0/1048633126 >> 172.21.15.113:0/1048633126 conn(0x7f608c0b1740 msgr2=0x7f608c06d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.825+0000 7f6092776700 1 -- 172.21.15.113:0/1048633126 shutdown_connections 2024-01-15T11:25:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.825+0000 7f6092776700 1 -- 172.21.15.113:0/1048633126 wait complete. 2024-01-15T11:25:22.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.829+0000 7f60b71a4700 1 -- 172.21.15.113:0/933031356 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f60b0074a20 con 0x7f60b0158a50 2024-01-15T11:25:22.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.830+0000 7f60ad7fa700 1 -- 172.21.15.113:0/933031356 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f609c057d00 con 0x7f60b0158a50 2024-01-15T11:25:22.829 INFO:teuthology.orchestra.run.smithi113.stdout:55834574854 2024-01-15T11:25:22.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.831+0000 7f84b8bc2700 1 -- 172.21.15.113:0/4257208812 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f84b4074b00 con 0x7f84b4072df0 2024-01-15T11:25:22.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.831+0000 7f84a2ffd700 1 -- 172.21.15.113:0/4257208812 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f84a8058320 con 0x7f84b4072df0 2024-01-15T11:25:22.830 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-15T11:25:22.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.834+0000 7f60b71a4700 1 -- 172.21.15.113:0/933031356 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f609406b420 msgr2=0x7f609406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.834+0000 7f60b71a4700 1 --2- 172.21.15.113:0/933031356 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f609406b420 0x7f609406d8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f60a4001600 tx=0x7f60a4007ad0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.834+0000 7f60b71a4700 1 -- 172.21.15.113:0/933031356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60b0158a50 msgr2=0x7f60b00b4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.834+0000 7f60b71a4700 1 --2- 172.21.15.113:0/933031356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60b0158a50 0x7f60b00b4a60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f609c00e810 tx=0x7f609c00ebd0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.835+0000 7f60b71a4700 1 -- 172.21.15.113:0/933031356 shutdown_connections 2024-01-15T11:25:22.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.835+0000 7f60b71a4700 1 --2- 172.21.15.113:0/933031356 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f609406b420 0x7f609406d8e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.835+0000 7f60b71a4700 1 --2- 172.21.15.113:0/933031356 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0162a30 0x7f60b0157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.835+0000 7f60b71a4700 1 --2- 172.21.15.113:0/933031356 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f60b0158670 0x7f60b00adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.835+0000 7f60b71a4700 1 --2- 172.21.15.113:0/933031356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60b0158a50 0x7f60b00b4a60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.835+0000 7f60b71a4700 1 -- 172.21.15.113:0/933031356 >> 172.21.15.113:0/933031356 conn(0x7f60b00b57b0 msgr2=0x7f60b006e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.835+0000 7f84b8bc2700 1 -- 172.21.15.113:0/4257208812 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f849406b350 msgr2=0x7f849406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.835+0000 7f84b8bc2700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f849406b350 0x7f849406d810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f84b40d0cc0 tx=0x7f84ac013000 comp rx=0 tx=0).stop 2024-01-15T11:25:22.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.835+0000 7f84b8bc2700 1 -- 172.21.15.113:0/4257208812 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f84b4072df0 msgr2=0x7f84b40aeeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.835+0000 7f84b8bc2700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f84b4072df0 0x7f84b40aeeb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f84a800c550 tx=0x7f84a800c860 comp rx=0 tx=0).stop 2024-01-15T11:25:22.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.835+0000 7f60b71a4700 1 -- 172.21.15.113:0/933031356 shutdown_connections 2024-01-15T11:25:22.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.835+0000 7f60b71a4700 1 -- 172.21.15.113:0/933031356 wait complete. 2024-01-15T11:25:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.838+0000 7f84b8bc2700 1 -- 172.21.15.113:0/4257208812 shutdown_connections 2024-01-15T11:25:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.838+0000 7f84b8bc2700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f849406b350 0x7f849406d810 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.838+0000 7f84b8bc2700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84b415ddf0 0x7f84b41f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.838+0000 7f84b8bc2700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f84b4072df0 0x7f84b40aeeb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.838+0000 7f84b8bc2700 1 --2- 172.21.15.113:0/4257208812 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84b40b0230 0x7f84b41e2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.838+0000 7f84b8bc2700 1 -- 172.21.15.113:0/4257208812 >> 172.21.15.113:0/4257208812 conn(0x7f84b40b57b0 msgr2=0x7f84b41e5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.841+0000 7f84b8bc2700 1 -- 172.21.15.113:0/4257208812 shutdown_connections 2024-01-15T11:25:22.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.841+0000 7f84b8bc2700 1 -- 172.21.15.113:0/4257208812 wait complete. 2024-01-15T11:25:22.840 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-15T11:25:22.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.841+0000 7fe115043700 1 -- 172.21.15.113:0/2944394662 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe0f40055a0 con 0x7fe110163480 2024-01-15T11:25:22.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.841+0000 7fe1077fe700 1 -- 172.21.15.113:0/2944394662 <== mon.1 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe100058000 con 0x7fe110163480 2024-01-15T11:25:22.843 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.7 2024-01-15T11:25:22.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.844+0000 7fe115043700 1 -- 172.21.15.113:0/2944394662 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe0fc06b5b0 msgr2=0x7fe0fc06da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.844+0000 7fe115043700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe0fc06b5b0 0x7fe0fc06da70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009c10 tx=0x7fe0f8010000 comp rx=0 tx=0).stop 2024-01-15T11:25:22.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.844+0000 7fe115043700 1 -- 172.21.15.113:0/2944394662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe110163480 msgr2=0x7fe110146470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.844+0000 7fe115043700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe110163480 0x7fe110146470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe10000e900 tx=0x7fe10000ec10 comp rx=0 tx=0).stop 2024-01-15T11:25:22.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.846+0000 7fe115043700 1 -- 172.21.15.113:0/2944394662 shutdown_connections 2024-01-15T11:25:22.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.846+0000 7fe115043700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe0fc06b5b0 0x7fe0fc06da70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.846+0000 7fe115043700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110163860 0x7fe11014fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.846+0000 7fe115043700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe1100b1e40 0x7fe11014d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.846+0000 7fe115043700 1 --2- 172.21.15.113:0/2944394662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe110163480 0x7fe110146470 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.846+0000 7fe115043700 1 -- 172.21.15.113:0/2944394662 >> 172.21.15.113:0/2944394662 conn(0x7fe1100b57b0 msgr2=0x7fe1101607d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.845 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-01-15T11:25:22.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.846+0000 7fe115043700 1 -- 172.21.15.113:0/2944394662 shutdown_connections 2024-01-15T11:25:22.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.846+0000 7fe115043700 1 -- 172.21.15.113:0/2944394662 wait complete. 2024-01-15T11:25:22.846 DEBUG:teuthology.parallel:result is None 2024-01-15T11:25:22.852 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.1 2024-01-15T11:25:22.856 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.3 2024-01-15T11:25:22.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.858+0000 7f21196e7700 1 -- 172.21.15.113:0/3504094886 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f20f40055a0 con 0x7f21140b1e40 2024-01-15T11:25:22.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.859+0000 7f20fb7fe700 1 -- 172.21.15.113:0/3504094886 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2108059900 con 0x7f21140b1e40 2024-01-15T11:25:22.857 INFO:teuthology.orchestra.run.smithi113.stdout:55834574852 2024-01-15T11:25:22.859 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2024-01-15T11:25:22.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.861+0000 7f21196e7700 1 -- 172.21.15.113:0/3504094886 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f210006b350 msgr2=0x7f210006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.861+0000 7f21196e7700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f210006b350 0x7f210006d810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f21141564b0 tx=0x7f2104009210 comp rx=0 tx=0).stop 2024-01-15T11:25:22.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.861+0000 7f21196e7700 1 -- 172.21.15.113:0/3504094886 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f21140b1e40 msgr2=0x7f21141669b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:22.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.861+0000 7f21196e7700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f21140b1e40 0x7f21141669b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2108015000 tx=0x7f2108004750 comp rx=0 tx=0).stop 2024-01-15T11:25:22.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.861+0000 7f21196e7700 1 -- 172.21.15.113:0/3504094886 shutdown_connections 2024-01-15T11:25:22.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.861+0000 7f21196e7700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f210006b350 0x7f210006d810 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.861+0000 7f21196e7700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2114163860 0x7f21141f3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.861+0000 7f21196e7700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f21140b1e40 0x7f21141669b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.861+0000 7f21196e7700 1 --2- 172.21.15.113:0/3504094886 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2114163480 0x7f2114155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:22.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.861+0000 7f21196e7700 1 -- 172.21.15.113:0/3504094886 >> 172.21.15.113:0/3504094886 conn(0x7f21140b57b0 msgr2=0x7f21141616c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:22.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.861+0000 7f21196e7700 1 -- 172.21.15.113:0/3504094886 shutdown_connections 2024-01-15T11:25:22.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:22.861+0000 7f21196e7700 1 -- 172.21.15.113:0/3504094886 wait complete. 2024-01-15T11:25:22.872 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.4 2024-01-15T11:25:23.822 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-15T11:25:23.824 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-15T11:25:23.844 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-15T11:25:23.853 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-15T11:25:23.856 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-15T11:25:23.860 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-15T11:25:23.872 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-15T11:25:24.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.013+0000 7f688a4d6700 1 Processor -- start 2024-01-15T11:25:24.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.013+0000 7f688a4d6700 1 -- start start 2024-01-15T11:25:24.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.014+0000 7f688a4d6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68840d0520 0x7f68840b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.014+0000 7f688a4d6700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f688415d610 0x7f68840b4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.014+0000 7f688a4d6700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68840720e0 con 0x7f68840714d0 2024-01-15T11:25:24.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.014+0000 7f688a4d6700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6884072d60 con 0x7f688415d610 2024-01-15T11:25:24.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.014+0000 7f688a4d6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688406de80 con 0x7f68840d0520 2024-01-15T11:25:24.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.014+0000 7f2c00186700 1 Processor -- start 2024-01-15T11:25:24.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.015+0000 7f6888a73700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f68840714d0 0x7f6884071930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:37564/0 (socket says 172.21.15.113:37564) 2024-01-15T11:25:24.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.015+0000 7f6888a73700 1 -- 172.21.15.113:0/2023964722 learned_addr learned my addr 172.21.15.113:0/2023964722 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:24.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.015+0000 7f2c00186700 1 -- start start 2024-01-15T11:25:24.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.015+0000 7f68837fe700 1 --2- 172.21.15.113:0/2023964722 >> v2:172.21.15.141:3300/0 conn(0x7f688415d610 0x7f68840b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.015+0000 7f6883fff700 1 --2- 172.21.15.113:0/2023964722 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68840d0520 0x7f68840b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.015+0000 7f6882ffd700 1 -- 172.21.15.113:0/2023964722 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521191193 0 0) 0x7f68840720e0 con 0x7f68840714d0 2024-01-15T11:25:24.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.015+0000 7f6882ffd700 1 -- 172.21.15.113:0/2023964722 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f686c0035b0 con 0x7f68840714d0 2024-01-15T11:25:24.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.016+0000 7f68837fe700 1 -- 172.21.15.113:0/2023964722 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68840d0520 msgr2=0x7f68840b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:24.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.016+0000 7f68837fe700 1 --2- 172.21.15.113:0/2023964722 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68840d0520 0x7f68840b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.016+0000 7f68837fe700 1 -- 172.21.15.113:0/2023964722 >> v1:172.21.15.113:6789/0 conn(0x7f68840714d0 legacy=0x7f6884071930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.016+0000 7f68837fe700 1 -- 172.21.15.113:0/2023964722 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68840723c0 con 0x7f688415d610 2024-01-15T11:25:24.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.016+0000 7f2c00186700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bf80d0d00 0x7f2bf80b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.016+0000 7f2c00186700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80d1190 con 0x7f2bf815e1e0 2024-01-15T11:25:24.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.016+0000 7f2c00186700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80d20b0 con 0x7f2bf80d0870 2024-01-15T11:25:24.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.016+0000 7f2c00186700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf80d3150 con 0x7f2bf80d0d00 2024-01-15T11:25:24.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.016+0000 7f2bfdf22700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f2bf815e1e0 0x7f2bf80d34c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:37570/0 (socket says 172.21.15.113:37570) 2024-01-15T11:25:24.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.016+0000 7f2bfdf22700 1 -- 172.21.15.113:0/3226878547 learned_addr learned my addr 172.21.15.113:0/3226878547 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:24.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.016+0000 7f2bfe723700 1 --2- 172.21.15.113:0/3226878547 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bf80d0d00 0x7f2bf80b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f68837fe700 1 --2- 172.21.15.113:0/2023964722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f688415d610 0x7f68840b4090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6878009870 tx=0x7f6878009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7de865fcd15fbb87 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f2bfcf20700 1 -- 172.21.15.113:0/3226878547 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704713959 0 0) 0x7f2bf80d1190 con 0x7f2bf815e1e0 2024-01-15T11:25:24.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f2bfcf20700 1 -- 172.21.15.113:0/3226878547 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be40035b0 con 0x7f2bf815e1e0 2024-01-15T11:25:24.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f2bfcf20700 1 -- 172.21.15.113:0/3226878547 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726317577 0 0) 0x7f2bf80d20b0 con 0x7f2bf80d0870 2024-01-15T11:25:24.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f2bfcf20700 1 -- 172.21.15.113:0/3226878547 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf80d1190 con 0x7f2bf80d0870 2024-01-15T11:25:24.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f6882ffd700 1 -- 172.21.15.113:0/2023964722 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c001670 con 0x7f688415d610 2024-01-15T11:25:24.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f6882ffd700 1 -- 172.21.15.113:0/2023964722 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6878004030 con 0x7f688415d610 2024-01-15T11:25:24.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f6882ffd700 1 -- 172.21.15.113:0/2023964722 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68780043c0 con 0x7f688415d610 2024-01-15T11:25:24.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.018+0000 7f688a4d6700 1 -- 172.21.15.113:0/2023964722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f688415d610 msgr2=0x7f68840b4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.018+0000 7f688a4d6700 1 --2- 172.21.15.113:0/2023964722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f688415d610 0x7f68840b4090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6878009870 tx=0x7f6878009b80 comp rx=0 tx=0).stop 2024-01-15T11:25:24.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f2bfe723700 1 -- 172.21.15.113:0/3226878547 >> v1:172.21.15.141:6789/0 conn(0x7f2bf80d0870 legacy=0x7f2bf80b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f2bfe723700 1 -- 172.21.15.113:0/3226878547 >> v1:172.21.15.113:6789/0 conn(0x7f2bf815e1e0 legacy=0x7f2bf80d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f2bfe723700 1 -- 172.21.15.113:0/3226878547 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf80d2390 con 0x7f2bf80d0d00 2024-01-15T11:25:24.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f2bfe723700 1 --2- 172.21.15.113:0/3226878547 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bf80d0d00 0x7f2bf80b7600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4009870 tx=0x7f2bf4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=113f1fa04972306 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f2bfcf20700 1 -- 172.21.15.113:0/3226878547 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4011830 con 0x7f2bf80d0d00 2024-01-15T11:25:24.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f2bfcf20700 1 -- 172.21.15.113:0/3226878547 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf4011990 con 0x7f2bf80d0d00 2024-01-15T11:25:24.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.017+0000 7f2bfcf20700 1 -- 172.21.15.113:0/3226878547 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4011d00 con 0x7f2bf80d0d00 2024-01-15T11:25:24.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.018+0000 7f688a4d6700 1 -- 172.21.15.113:0/2023964722 shutdown_connections 2024-01-15T11:25:24.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.018+0000 7f688a4d6700 1 --2- 172.21.15.113:0/2023964722 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68840d0520 0x7f68840b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.018+0000 7f688a4d6700 1 --2- 172.21.15.113:0/2023964722 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f688415d610 0x7f68840b4090 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.018+0000 7f688a4d6700 1 -- 172.21.15.113:0/2023964722 >> 172.21.15.113:0/2023964722 conn(0x7f68840cfa30 msgr2=0x7f68840cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.018+0000 7f2c00186700 1 -- 172.21.15.113:0/3226878547 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bf80d0d00 msgr2=0x7f2bf80b7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.018+0000 7f2c00186700 1 --2- 172.21.15.113:0/3226878547 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bf80d0d00 0x7f2bf80b7600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4009870 tx=0x7f2bf4009b80 comp rx=0 tx=0).stop 2024-01-15T11:25:24.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.018+0000 7f2c00186700 1 -- 172.21.15.113:0/3226878547 shutdown_connections 2024-01-15T11:25:24.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.018+0000 7f2c00186700 1 --2- 172.21.15.113:0/3226878547 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bf80d0d00 0x7f2bf80b7600 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.018+0000 7f2c00186700 1 -- 172.21.15.113:0/3226878547 >> 172.21.15.113:0/3226878547 conn(0x7f2bf80b5b50 msgr2=0x7f2bf80b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.019+0000 7f688a4d6700 1 -- 172.21.15.113:0/2023964722 shutdown_connections 2024-01-15T11:25:24.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.019+0000 7f2c00186700 1 -- 172.21.15.113:0/3226878547 shutdown_connections 2024-01-15T11:25:24.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.019+0000 7f2c00186700 1 -- 172.21.15.113:0/3226878547 wait complete. 2024-01-15T11:25:24.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.019+0000 7f688a4d6700 1 -- 172.21.15.113:0/2023964722 wait complete. 2024-01-15T11:25:24.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.021+0000 7f2c00186700 1 Processor -- start 2024-01-15T11:25:24.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.021+0000 7f688a4d6700 1 Processor -- start 2024-01-15T11:25:24.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.021+0000 7f2c00186700 1 -- start start 2024-01-15T11:25:24.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f2c00186700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf80d0870 0x7f2bf81f8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f2c00186700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bf80d0d00 0x7f2bf815ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f2c00186700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bf815e1e0 0x7f2bf815b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f2c00186700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf815f790 con 0x7f2bf815e1e0 2024-01-15T11:25:24.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f2c00186700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf81f8a30 con 0x7f2bf80d0870 2024-01-15T11:25:24.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f2c00186700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf81f8d10 con 0x7f2bf80d0d00 2024-01-15T11:25:24.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f2bfe723700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bf815e1e0 0x7f2bf815b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f2bfe723700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bf815e1e0 0x7f2bf815b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45772/0 (socket says 172.21.15.113:45772) 2024-01-15T11:25:24.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f2bfe723700 1 -- 172.21.15.113:0/1142688803 learned_addr learned my addr 172.21.15.113:0/1142688803 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:24.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f2bfd721700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bf80d0d00 0x7f2bf815ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f2bfdf22700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf80d0870 0x7f2bf81f8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.021+0000 7f688a4d6700 1 -- start start 2024-01-15T11:25:24.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f688a4d6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68840714d0 0x7f6884153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f688a4d6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68840d0520 0x7f688413e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f688a4d6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f688415d610 0x7f6884152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f688a4d6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68841f8650 con 0x7f688415d610 2024-01-15T11:25:24.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f688a4d6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688413eb10 con 0x7f68840d0520 2024-01-15T11:25:24.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.022+0000 7f688a4d6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68841f7d70 con 0x7f68840714d0 2024-01-15T11:25:24.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f6888a73700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f688415d610 0x7f6884152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f6888a73700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f688415d610 0x7f6884152260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45776/0 (socket says 172.21.15.113:45776) 2024-01-15T11:25:24.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f6888a73700 1 -- 172.21.15.113:0/1264768226 learned_addr learned my addr 172.21.15.113:0/1264768226 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:24.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f68837fe700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68840d0520 0x7f688413e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f2bfd721700 1 -- 172.21.15.113:0/1142688803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf80d0870 msgr2=0x7f2bf81f8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f2bfd721700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf80d0870 0x7f2bf81f8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f2bfd721700 1 -- 172.21.15.113:0/1142688803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bf815e1e0 msgr2=0x7f2bf815b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f2bfd721700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bf815e1e0 0x7f2bf815b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f2bfd721700 1 -- 172.21.15.113:0/1142688803 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf40094a0 con 0x7f2bf80d0d00 2024-01-15T11:25:24.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f2bfd721700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bf80d0d00 0x7f2bf815ff30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000cbe0 tx=0x7f2bf000cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f2bee7fc700 1 -- 172.21.15.113:0/1142688803 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0003c00 con 0x7f2bf80d0d00 2024-01-15T11:25:24.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f6883fff700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68840714d0 0x7f6884153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f6888a73700 1 -- 172.21.15.113:0/1264768226 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68840714d0 msgr2=0x7f6884153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.023+0000 7f2bee7fc700 1 -- 172.21.15.113:0/1142688803 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf0003d60 con 0x7f2bf80d0d00 2024-01-15T11:25:24.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.024+0000 7f2bee7fc700 1 -- 172.21.15.113:0/1142688803 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf000aae0 con 0x7f2bf80d0d00 2024-01-15T11:25:24.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.024+0000 7f2c00186700 1 -- 172.21.15.113:0/1142688803 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf81f8ff0 con 0x7f2bf80d0d00 2024-01-15T11:25:24.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.024+0000 7f2c00186700 1 -- 172.21.15.113:0/1142688803 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf81f8740 con 0x7f2bf80d0d00 2024-01-15T11:25:24.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.024+0000 7f6888a73700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68840714d0 0x7f6884153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.024+0000 7f6888a73700 1 -- 172.21.15.113:0/1264768226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68840d0520 msgr2=0x7f688413e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.024+0000 7f6888a73700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68840d0520 0x7f688413e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.024+0000 7f6888a73700 1 -- 172.21.15.113:0/1264768226 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68780094a0 con 0x7f688415d610 2024-01-15T11:25:24.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.024+0000 7f6883fff700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68840714d0 0x7f6884153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:24.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.024+0000 7f6888a73700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f688415d610 0x7f6884152260 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f687c00c7e0 tx=0x7f687c00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.025+0000 7f6880ff9700 1 -- 172.21.15.113:0/1264768226 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c007560 con 0x7f688415d610 2024-01-15T11:25:24.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.025+0000 7f6880ff9700 1 -- 172.21.15.113:0/1264768226 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687c007700 con 0x7f688415d610 2024-01-15T11:25:24.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.026+0000 7f2c00186700 1 -- 172.21.15.113:0/1142688803 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf80751e0 con 0x7f2bf80d0d00 2024-01-15T11:25:24.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.026+0000 7f2bee7fc700 1 -- 172.21.15.113:0/1142688803 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2bf000ac40 con 0x7f2bf80d0d00 2024-01-15T11:25:24.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.025+0000 7f6880ff9700 1 -- 172.21.15.113:0/1264768226 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c004ab0 con 0x7f688415d610 2024-01-15T11:25:24.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.025+0000 7f688a4d6700 1 -- 172.21.15.113:0/1264768226 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6884152d90 con 0x7f688415d610 2024-01-15T11:25:24.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.026+0000 7f688a4d6700 1 -- 172.21.15.113:0/1264768226 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68841f7f90 con 0x7f688415d610 2024-01-15T11:25:24.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.026+0000 7f2bee7fc700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2bdc06b350 0x7f2bdc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.026+0000 7f2bee7fc700 1 -- 172.21.15.113:0/1142688803 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2bf0089c50 con 0x7f2bf80d0d00 2024-01-15T11:25:24.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.027+0000 7f688a4d6700 1 -- 172.21.15.113:0/1264768226 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68841f8790 con 0x7f688415d610 2024-01-15T11:25:24.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.027+0000 7f2bfdf22700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2bdc06b350 0x7f2bdc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.028+0000 7f2bfdf22700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2bdc06b350 0x7f2bdc06d810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2be8007fd0 tx=0x7f2be8007dc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.028+0000 7f6880ff9700 1 -- 172.21.15.113:0/1264768226 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f687c01c3b0 con 0x7f688415d610 2024-01-15T11:25:24.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.028+0000 7f6880ff9700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f687006b3a0 0x7f687006d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.029+0000 7f6880ff9700 1 -- 172.21.15.113:0/1264768226 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f687c088f60 con 0x7f688415d610 2024-01-15T11:25:24.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.030+0000 7f6883fff700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f687006b3a0 0x7f687006d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.030+0000 7f6883fff700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f687006b3a0 0x7f687006d860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f68840cf5a0 tx=0x7f687400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.033+0000 7f2bee7fc700 1 -- 172.21.15.113:0/1142688803 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2bf008f020 con 0x7f2bf80d0d00 2024-01-15T11:25:24.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.033+0000 7f6880ff9700 1 -- 172.21.15.113:0/1264768226 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f687c04de90 con 0x7f688415d610 2024-01-15T11:25:24.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.041+0000 7fa4b3fa2700 1 Processor -- start 2024-01-15T11:25:24.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.041+0000 7fa4b3fa2700 1 -- start start 2024-01-15T11:25:24.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.042+0000 7fa4b3fa2700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa4ac0ae4e0 0x7fa4ac06c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.042+0000 7fa4b3fa2700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4ac0b46e0 con 0x7fa4ac06f360 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.042+0000 7fa4b3fa2700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4ac0b49c0 con 0x7fa4ac162240 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.042+0000 7fa4b3fa2700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ac0b4ca0 con 0x7fa4ac0ae4e0 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.043+0000 7fa4b153d700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa4ac0ae4e0 0x7fa4ac06c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.043+0000 7fa4b153d700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa4ac0ae4e0 0x7fa4ac06c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51588/0 (socket says 172.21.15.113:51588) 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.043+0000 7fa4b153d700 1 -- 172.21.15.113:0/2694626419 learned_addr learned my addr 172.21.15.113:0/2694626419 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.043+0000 7fa4b153d700 1 -- 172.21.15.113:0/2694626419 >> v1:172.21.15.141:6789/0 conn(0x7fa4ac162240 legacy=0x7fa4ac0afed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.043+0000 7fa4b153d700 1 -- 172.21.15.113:0/2694626419 >> v1:172.21.15.113:6789/0 conn(0x7fa4ac06f360 legacy=0x7fa4ac165f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.043+0000 7fa4b153d700 1 -- 172.21.15.113:0/2694626419 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac0b4f80 con 0x7fa4ac0ae4e0 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.043+0000 7fa4b153d700 1 --2- 172.21.15.113:0/2694626419 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ac0ae4e0 0x7fa4ac06c000 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa49c0098f0 tx=0x7fa49c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ae7c46b8084862 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.044+0000 7fa4b0d3c700 1 -- 172.21.15.113:0/2694626419 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c004230 con 0x7fa4ac0ae4e0 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.044+0000 7fa4b0d3c700 1 -- 172.21.15.113:0/2694626419 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c004390 con 0x7fa4ac0ae4e0 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.044+0000 7fa4b0d3c700 1 -- 172.21.15.113:0/2694626419 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0036a0 con 0x7fa4ac0ae4e0 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.045+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/2694626419 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ac0ae4e0 msgr2=0x7fa4ac06c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.045+0000 7fa4b3fa2700 1 --2- 172.21.15.113:0/2694626419 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ac0ae4e0 0x7fa4ac06c000 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa49c0098f0 tx=0x7fa49c009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:24.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.045+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/2694626419 shutdown_connections 2024-01-15T11:25:24.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.045+0000 7fa4b3fa2700 1 --2- 172.21.15.113:0/2694626419 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ac0ae4e0 0x7fa4ac06c000 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.045+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/2694626419 >> 172.21.15.113:0/2694626419 conn(0x7fa4ac071b90 msgr2=0x7fa4ac071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.045+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/2694626419 shutdown_connections 2024-01-15T11:25:24.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.045+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/2694626419 wait complete. 2024-01-15T11:25:24.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.047+0000 7fa4b3fa2700 1 Processor -- start 2024-01-15T11:25:24.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.047+0000 7fa4b3fa2700 1 -- start start 2024-01-15T11:25:24.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.048+0000 7fa4b3fa2700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ac06f360 0x7fa4ac162920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.048+0000 7fa4b3fa2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4ac0ae4e0 0x7fa4ac161580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.048+0000 7fa4b3fa2700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4ac162240 0x7fa4ac074200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.048+0000 7fa4b3fa2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac165830 con 0x7fa4ac0ae4e0 2024-01-15T11:25:24.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.048+0000 7fa4b3fa2700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac161ac0 con 0x7fa4ac06f360 2024-01-15T11:25:24.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.048+0000 7fa4b3fa2700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac163830 con 0x7fa4ac162240 2024-01-15T11:25:24.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.049+0000 7fa4b153d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4ac0ae4e0 0x7fa4ac161580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.049+0000 7fa4b253f700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4ac162240 0x7fa4ac074200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.049+0000 7fa4b153d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4ac0ae4e0 0x7fa4ac161580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45784/0 (socket says 172.21.15.113:45784) 2024-01-15T11:25:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.049+0000 7fa4b153d700 1 -- 172.21.15.113:0/503598552 learned_addr learned my addr 172.21.15.113:0/503598552 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.049+0000 7fa4b253f700 1 -- 172.21.15.113:0/503598552 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ac06f360 msgr2=0x7fa4ac162920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T11:25:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.049+0000 7fa4b253f700 1 --2- 172.21.15.113:0/503598552 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ac06f360 0x7fa4ac162920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.049+0000 7fa4b253f700 1 -- 172.21.15.113:0/503598552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4ac0ae4e0 msgr2=0x7fa4ac161580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.049+0000 7fa4b253f700 1 --2- 172.21.15.113:0/503598552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4ac0ae4e0 0x7fa4ac161580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.049+0000 7fa4b253f700 1 -- 172.21.15.113:0/503598552 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c009550 con 0x7fa4ac162240 2024-01-15T11:25:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.049+0000 7fa4b253f700 1 --2- 172.21.15.113:0/503598552 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4ac162240 0x7fa4ac074200 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400c780 tx=0x7fa4a400ca90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.050+0000 7fa4a27fc700 1 -- 172.21.15.113:0/503598552 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4003e10 con 0x7fa4ac162240 2024-01-15T11:25:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.050+0000 7fa4a27fc700 1 -- 172.21.15.113:0/503598552 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a40050f0 con 0x7fa4ac162240 2024-01-15T11:25:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.050+0000 7fa4a27fc700 1 -- 172.21.15.113:0/503598552 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a40054a0 con 0x7fa4ac162240 2024-01-15T11:25:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.050+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/503598552 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac162f90 con 0x7fa4ac162240 2024-01-15T11:25:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.050+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/503598552 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac163550 con 0x7fa4ac162240 2024-01-15T11:25:24.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.052+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/503598552 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ac0753f0 con 0x7fa4ac162240 2024-01-15T11:25:24.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.052+0000 7fa4a27fc700 1 -- 172.21.15.113:0/503598552 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa4a4005600 con 0x7fa4ac162240 2024-01-15T11:25:24.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.053+0000 7fa4a27fc700 1 --2- 172.21.15.113:0/503598552 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa49006b420 0x7fa49006d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.053+0000 7fa4a27fc700 1 -- 172.21.15.113:0/503598552 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4a40890f0 con 0x7fa4ac162240 2024-01-15T11:25:24.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.053+0000 7fa4b1d3e700 1 --2- 172.21.15.113:0/503598552 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa49006b420 0x7fa49006d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.054+0000 7fa4b1d3e700 1 --2- 172.21.15.113:0/503598552 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa49006b420 0x7fa49006d8e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009670 tx=0x7fa4a8006bd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.055+0000 7fc3674d0700 1 Processor -- start 2024-01-15T11:25:24.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.056+0000 7fc3674d0700 1 -- start start 2024-01-15T11:25:24.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.056+0000 7fc3674d0700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc360162240 0x7fc3600afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.056+0000 7fc3674d0700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc36006c710 0x7fc360165f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.056+0000 7fc3674d0700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3600b46e0 con 0x7fc3600ae4e0 2024-01-15T11:25:24.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.056+0000 7fc3674d0700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3600b49c0 con 0x7fc360162240 2024-01-15T11:25:24.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.056+0000 7fc3674d0700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3600b4ca0 con 0x7fc36006c710 2024-01-15T11:25:24.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.057+0000 7fc36526c700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc360162240 0x7fc3600afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.057+0000 7fc36526c700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc360162240 0x7fc3600afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51596/0 (socket says 172.21.15.113:51596) 2024-01-15T11:25:24.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.057+0000 7fc36526c700 1 -- 172.21.15.113:0/2424996846 learned_addr learned my addr 172.21.15.113:0/2424996846 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:24.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.057+0000 7fc36526c700 1 -- 172.21.15.113:0/2424996846 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc36006c710 msgr2=0x7fc360165f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T11:25:24.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.057+0000 7fc36526c700 1 --2- 172.21.15.113:0/2424996846 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc36006c710 0x7fc360165f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.057+0000 7fc36526c700 1 -- 172.21.15.113:0/2424996846 >> v1:172.21.15.141:6789/0 conn(0x7fc3600ae4e0 legacy=0x7fc36006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:24.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.057+0000 7fc36526c700 1 -- 172.21.15.113:0/2424996846 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3600b4f80 con 0x7fc360162240 2024-01-15T11:25:24.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.057+0000 7fa4a27fc700 1 -- 172.21.15.113:0/503598552 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4a4054080 con 0x7fa4ac162240 2024-01-15T11:25:24.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.058+0000 7fc36526c700 1 --2- 172.21.15.113:0/2424996846 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc360162240 0x7fc3600afed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0098f0 tx=0x7fc35c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94947d21eef6f5c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.058+0000 7fc357fff700 1 -- 172.21.15.113:0/2424996846 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c004230 con 0x7fc360162240 2024-01-15T11:25:24.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.058+0000 7fc357fff700 1 -- 172.21.15.113:0/2424996846 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c004390 con 0x7fc360162240 2024-01-15T11:25:24.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.058+0000 7fc357fff700 1 -- 172.21.15.113:0/2424996846 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0036a0 con 0x7fc360162240 2024-01-15T11:25:24.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.058+0000 7fc3674d0700 1 -- 172.21.15.113:0/2424996846 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc360162240 msgr2=0x7fc3600afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.058+0000 7fc3674d0700 1 --2- 172.21.15.113:0/2424996846 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc360162240 0x7fc3600afed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0098f0 tx=0x7fc35c009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:24.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.059+0000 7fc3674d0700 1 -- 172.21.15.113:0/2424996846 shutdown_connections 2024-01-15T11:25:24.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.059+0000 7fc3674d0700 1 --2- 172.21.15.113:0/2424996846 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc36006c710 0x7fc360165f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.059+0000 7fc3674d0700 1 --2- 172.21.15.113:0/2424996846 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc360162240 0x7fc3600afed0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.059+0000 7fc3674d0700 1 -- 172.21.15.113:0/2424996846 >> 172.21.15.113:0/2424996846 conn(0x7fc360071b90 msgr2=0x7fc360071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.059+0000 7fc3674d0700 1 -- 172.21.15.113:0/2424996846 shutdown_connections 2024-01-15T11:25:24.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.062+0000 7fc3674d0700 1 -- 172.21.15.113:0/2424996846 wait complete. 2024-01-15T11:25:24.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.063+0000 7fc3674d0700 1 Processor -- start 2024-01-15T11:25:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.067+0000 7fc3674d0700 1 -- start start 2024-01-15T11:25:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.067+0000 7fc3674d0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc36006c710 0x7fc3600cde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.067+0000 7fc3674d0700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc360162240 0x7fc36014a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.067+0000 7fc3674d0700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc36009e5d0 0x7fc3600cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.067+0000 7fc3674d0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3601496f0 con 0x7fc36006c710 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.067+0000 7fc3674d0700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc360148ae0 con 0x7fc360162240 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.067+0000 7fc3674d0700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc360148dc0 con 0x7fc36009e5d0 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.068+0000 7fc365a6d700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc36009e5d0 0x7fc3600cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.068+0000 7fc365a6d700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc36009e5d0 0x7fc3600cf9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:47488/0 (socket says 172.21.15.113:47488) 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.068+0000 7fc365a6d700 1 -- 172.21.15.113:0/1262110825 learned_addr learned my addr 172.21.15.113:0/1262110825 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.068+0000 7fc364a6b700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc360162240 0x7fc36014a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.068+0000 7fc365a6d700 1 -- 172.21.15.113:0/1262110825 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc360162240 msgr2=0x7fc36014a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.068+0000 7fc365a6d700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc360162240 0x7fc36014a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.068+0000 7fc365a6d700 1 -- 172.21.15.113:0/1262110825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc36006c710 msgr2=0x7fc3600cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.068+0000 7fc365a6d700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc36006c710 0x7fc3600cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.068+0000 7fc365a6d700 1 -- 172.21.15.113:0/1262110825 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c009550 con 0x7fc36009e5d0 2024-01-15T11:25:24.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.068+0000 7fc365a6d700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc36009e5d0 0x7fc3600cf9e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc35800b780 tx=0x7fc35800bb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.068+0000 7fc355ffb700 1 -- 172.21.15.113:0/1262110825 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35800f840 con 0x7fc36009e5d0 2024-01-15T11:25:24.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.069+0000 7fc355ffb700 1 -- 172.21.15.113:0/1262110825 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35800f9a0 con 0x7fc36009e5d0 2024-01-15T11:25:24.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.069+0000 7fc355ffb700 1 -- 172.21.15.113:0/1262110825 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35800fd50 con 0x7fc36009e5d0 2024-01-15T11:25:24.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.072+0000 7fc3674d0700 1 -- 172.21.15.113:0/1262110825 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3601490a0 con 0x7fc36009e5d0 2024-01-15T11:25:24.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.072+0000 7fc3674d0700 1 -- 172.21.15.113:0/1262110825 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3600ce3c0 con 0x7fc36009e5d0 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.064+0000 7fc46a114700 1 Processor -- start 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.064+0000 7fc46a114700 1 -- start start 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.066+0000 7fc46a114700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc4640b4260 0x7fc46406efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.066+0000 7fc46a114700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46408bdb0 con 0x7fc464072bb0 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.066+0000 7fc46a114700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46406ce30 con 0x7fc46416ae60 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.066+0000 7fc46a114700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc46406d110 con 0x7fc4640b4260 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.066+0000 7fc4637fe700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc46416ae60 0x7fc4640b3b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:35324/0 (socket says 172.21.15.113:35324) 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.066+0000 7fc4637fe700 1 -- 172.21.15.113:0/3823945512 learned_addr learned my addr 172.21.15.113:0/3823945512 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.066+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686500577 0 0) 0x7fc46406ce30 con 0x7fc46416ae60 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc44c003a00 con 0x7fc46416ae60 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132667150 0 0) 0x7fc46408bdb0 con 0x7fc464072bb0 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc46406ce30 con 0x7fc464072bb0 2024-01-15T11:25:24.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.069+0000 7fc463fff700 1 --2- 172.21.15.113:0/3823945512 >> v2:172.21.15.113:3300/0 conn(0x7fc4640b4260 0x7fc46406efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7f14bd28b700 1 Processor -- start 2024-01-15T11:25:24.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7fc3674d0700 1 -- 172.21.15.113:0/1262110825 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3400052f0 con 0x7fc36009e5d0 2024-01-15T11:25:24.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7fc355ffb700 1 -- 172.21.15.113:0/1262110825 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc3580203f0 con 0x7fc36009e5d0 2024-01-15T11:25:24.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc355ffb700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc34c06b5b0 0x7fc34c06da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc355ffb700 1 -- 172.21.15.113:0/1262110825 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc358089f10 con 0x7fc36009e5d0 2024-01-15T11:25:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 <== mon.1 v1:172.21.15.141:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 136285925 0 0) 0x7fc44c003a00 con 0x7fc46416ae60 2024-01-15T11:25:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 >> v2:172.21.15.113:3300/0 conn(0x7fc4640b4260 msgr2=0x7fc46406efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7fc4627fc700 1 --2- 172.21.15.113:0/3823945512 >> v2:172.21.15.113:3300/0 conn(0x7fc4640b4260 0x7fc46406efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 >> v1:172.21.15.113:6789/0 conn(0x7fc464072bb0 legacy=0x7fc464070d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 --> v1:172.21.15.141:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46406d3f0 con 0x7fc46416ae60 2024-01-15T11:25:24.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 <== mon.1 v1:172.21.15.141:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fc458001670 con 0x7fc46416ae60 2024-01-15T11:25:24.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 >> v1:172.21.15.141:6789/0 conn(0x7fc46416ae60 legacy=0x7fc4640b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4627fc700 1 --2- 172.21.15.113:0/3823945512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 0x7fc44c0050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4627fc700 1 --2- 172.21.15.113:0/3823945512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc44c0056a0 0x7fc44c003c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4627fc700 1 --2- 172.21.15.113:0/3823945512 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc44c0041d0 0x7fc44c004680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.074+0000 7f14bd28b700 1 -- start start 2024-01-15T11:25:24.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46406d110 con 0x7fc44c0056a0 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c00c200 con 0x7fc44c004ca0 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c00c3c0 con 0x7fc44c0041d0 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4637fe700 1 --2- 172.21.15.113:0/3823945512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 0x7fc44c0050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4637fe700 1 -- 172.21.15.113:0/3823945512 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc44c0041d0 msgr2=0x7fc44c004680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4637fe700 1 --2- 172.21.15.113:0/3823945512 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc44c0041d0 0x7fc44c004680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4637fe700 1 -- 172.21.15.113:0/3823945512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc44c0056a0 msgr2=0x7fc44c003c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4637fe700 1 --2- 172.21.15.113:0/3823945512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc44c0056a0 0x7fc44c003c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4637fe700 1 -- 172.21.15.113:0/3823945512 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46406d3f0 con 0x7fc44c004ca0 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.075+0000 7fc4637fe700 1 --2- 172.21.15.113:0/3823945512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 0x7fc44c0050a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc45801d2a0 tx=0x7fc44c005cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12bb3296b4fff7de server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.076+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458027070 con 0x7fc44c004ca0 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.076+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45801dbc0 con 0x7fc44c004ca0 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.076+0000 7fc4627fc700 1 -- 172.21.15.113:0/3823945512 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458017980 con 0x7fc44c004ca0 2024-01-15T11:25:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.076+0000 7fc463fff700 1 -- 172.21.15.113:0/3823945512 reap_dead start 2024-01-15T11:25:24.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.076+0000 7f14bd28b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b8163480 0x7f14b80b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.076+0000 7f14bd28b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f14b8163860 0x7f14b80b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.076+0000 7f14bd28b700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b80b2e80 con 0x7f14b80b1e40 2024-01-15T11:25:24.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.076+0000 7f14bd28b700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b80b29c0 con 0x7f14b8163860 2024-01-15T11:25:24.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.076+0000 7f14bd28b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b8164510 con 0x7f14b8163480 2024-01-15T11:25:24.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.076+0000 7f14b6ffd700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b8163480 0x7f14b80b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.076+0000 7f14b6ffd700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b8163480 0x7f14b80b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:47490/0 (socket says 172.21.15.113:47490) 2024-01-15T11:25:24.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.076+0000 7f14b6ffd700 1 -- 172.21.15.113:0/2097694994 learned_addr learned my addr 172.21.15.113:0/2097694994 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:24.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.077+0000 7fc46a114700 1 -- 172.21.15.113:0/3823945512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 msgr2=0x7fc44c0050a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.077+0000 7fc46a114700 1 --2- 172.21.15.113:0/3823945512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 0x7fc44c0050a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc45801d2a0 tx=0x7fc44c005cd0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.077+0000 7f14b6ffd700 1 -- 172.21.15.113:0/2097694994 >> v2:172.21.15.113:3300/0 conn(0x7f14b8163860 msgr2=0x7f14b80b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:24.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.077+0000 7f14b6ffd700 1 --2- 172.21.15.113:0/2097694994 >> v2:172.21.15.113:3300/0 conn(0x7f14b8163860 0x7f14b80b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.077+0000 7f14b6ffd700 1 -- 172.21.15.113:0/2097694994 >> v1:172.21.15.141:6789/0 conn(0x7f14b80b1e40 legacy=0x7f14b80b22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.077+0000 7f14b6ffd700 1 -- 172.21.15.113:0/2097694994 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b8160610 con 0x7f14b8163480 2024-01-15T11:25:24.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.077+0000 7fc46a114700 1 -- 172.21.15.113:0/3823945512 shutdown_connections 2024-01-15T11:25:24.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.077+0000 7fc46a114700 1 --2- 172.21.15.113:0/3823945512 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 0x7fc44c0050a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.077+0000 7fc46a114700 1 -- 172.21.15.113:0/3823945512 >> 172.21.15.113:0/3823945512 conn(0x7fc4640cfa30 msgr2=0x7fc4640cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.077+0000 7fc46a114700 1 -- 172.21.15.113:0/3823945512 shutdown_connections 2024-01-15T11:25:24.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.077+0000 7fc46a114700 1 -- 172.21.15.113:0/3823945512 wait complete. 2024-01-15T11:25:24.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.077+0000 7f14b6ffd700 1 --2- 172.21.15.113:0/2097694994 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b8163480 0x7f14b80b01a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f14a80098f0 tx=0x7f14a8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99286bf3f4ef792d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.078+0000 7fc36526c700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc34c06b5b0 0x7fc34c06da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.078+0000 7fc36526c700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc34c06b5b0 0x7fc34c06da70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc360149fd0 tx=0x7fc35c010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.079+0000 7fc46a114700 1 Processor -- start 2024-01-15T11:25:24.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.079+0000 7fc46a114700 1 -- start start 2024-01-15T11:25:24.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.079+0000 7f14b5ffb700 1 -- 172.21.15.113:0/2097694994 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8004230 con 0x7f14b8163480 2024-01-15T11:25:24.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.080+0000 7f14b5ffb700 1 -- 172.21.15.113:0/2097694994 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a8004390 con 0x7f14b8163480 2024-01-15T11:25:24.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.080+0000 7f14b5ffb700 1 -- 172.21.15.113:0/2097694994 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a80036a0 con 0x7f14b8163480 2024-01-15T11:25:24.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.080+0000 7fc46a114700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 0x7fc4640ac720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.080+0000 7fc46a114700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc4640cf1b0 0x7fc4640aac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.080+0000 7fc46a114700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4640ab150 0x7fc464156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.080+0000 7fc46a114700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4640acc60 con 0x7fc4640ab150 2024-01-15T11:25:24.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.080+0000 7fc46a114700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46408b210 con 0x7fc44c004ca0 2024-01-15T11:25:24.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.080+0000 7fc46a114700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4640aa780 con 0x7fc4640cf1b0 2024-01-15T11:25:24.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc4637fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 0x7fc4640ac720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc4637fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 0x7fc4640ac720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51628/0 (socket says 172.21.15.113:51628) 2024-01-15T11:25:24.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc4637fe700 1 -- 172.21.15.113:0/569269716 learned_addr learned my addr 172.21.15.113:0/569269716 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:24.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc355ffb700 1 -- 172.21.15.113:0/1262110825 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc358054d60 con 0x7fc36009e5d0 2024-01-15T11:25:24.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.080+0000 7f14bd28b700 1 -- 172.21.15.113:0/2097694994 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b8163480 msgr2=0x7f14b80b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.080+0000 7f14bd28b700 1 --2- 172.21.15.113:0/2097694994 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b8163480 0x7f14b80b01a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f14a80098f0 tx=0x7f14a8009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:24.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc463fff700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4640ab150 0x7fc464156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc462ffd700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc4640cf1b0 0x7fc4640aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc4637fe700 1 -- 172.21.15.113:0/569269716 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc4640cf1b0 msgr2=0x7fc4640aac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc4637fe700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc4640cf1b0 0x7fc4640aac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc4637fe700 1 -- 172.21.15.113:0/569269716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4640ab150 msgr2=0x7fc464156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc4637fe700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4640ab150 0x7fc464156e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc4637fe700 1 -- 172.21.15.113:0/569269716 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4640aa450 con 0x7fc44c004ca0 2024-01-15T11:25:24.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc4637fe700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 0x7fc4640ac720 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00c390 tx=0x7fc458001930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7f14bd28b700 1 -- 172.21.15.113:0/2097694994 shutdown_connections 2024-01-15T11:25:24.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7f14bd28b700 1 --2- 172.21.15.113:0/2097694994 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b8163480 0x7f14b80b01a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7f14bd28b700 1 --2- 172.21.15.113:0/2097694994 >> v2:172.21.15.113:3300/0 conn(0x7f14b8163860 0x7f14b80b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7f14bd28b700 1 -- 172.21.15.113:0/2097694994 >> 172.21.15.113:0/2097694994 conn(0x7f14b80b57b0 msgr2=0x7f14b80b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7f14bd28b700 1 -- 172.21.15.113:0/2097694994 shutdown_connections 2024-01-15T11:25:24.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.081+0000 7fc44bfff700 1 -- 172.21.15.113:0/569269716 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458027040 con 0x7fc44c004ca0 2024-01-15T11:25:24.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.082+0000 7fc44bfff700 1 -- 172.21.15.113:0/569269716 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc458017d90 con 0x7fc44c004ca0 2024-01-15T11:25:24.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.082+0000 7fc44bfff700 1 -- 172.21.15.113:0/569269716 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458001bd0 con 0x7fc44c004ca0 2024-01-15T11:25:24.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.082+0000 7f14bd28b700 1 -- 172.21.15.113:0/2097694994 wait complete. 2024-01-15T11:25:24.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.084+0000 7f14bd28b700 1 Processor -- start 2024-01-15T11:25:24.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.082+0000 7fc46a114700 1 -- 172.21.15.113:0/569269716 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4640c49b0 con 0x7fc44c004ca0 2024-01-15T11:25:24.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.082+0000 7fc46a114700 1 -- 172.21.15.113:0/569269716 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4640c4f70 con 0x7fc44c004ca0 2024-01-15T11:25:24.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.084+0000 7f14bd28b700 1 -- start start 2024-01-15T11:25:24.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.085+0000 7f14bd28b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b80b1e40 0x7f14b80ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.085+0000 7f14bd28b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14b8163480 0x7f14b80d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.085+0000 7f14bd28b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14b8163860 0x7f14b8148190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.085+0000 7f14bd28b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b815bb10 con 0x7f14b8163860 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.085+0000 7f14bd28b700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b80b3ae0 con 0x7f14b8163480 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.085+0000 7f14bd28b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b80b3dc0 con 0x7f14b80b1e40 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.086+0000 7f14b67fc700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14b8163480 0x7f14b80d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.086+0000 7f14b67fc700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14b8163480 0x7f14b80d05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:51642/0 (socket says 172.21.15.113:51642) 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.086+0000 7f14b67fc700 1 -- 172.21.15.113:0/493499869 learned_addr learned my addr 172.21.15.113:0/493499869 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.086+0000 7f14b6ffd700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b80b1e40 0x7f14b80ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.086+0000 7f14b77fe700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14b8163860 0x7f14b8148190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.086+0000 7f14b6ffd700 1 -- 172.21.15.113:0/493499869 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14b8163480 msgr2=0x7f14b80d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.086+0000 7f14b6ffd700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14b8163480 0x7f14b80d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.087+0000 7f14b6ffd700 1 -- 172.21.15.113:0/493499869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14b8163860 msgr2=0x7f14b8148190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.087+0000 7f14b6ffd700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14b8163860 0x7f14b8148190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.087+0000 7f14b6ffd700 1 -- 172.21.15.113:0/493499869 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a8009550 con 0x7f14b80b1e40 2024-01-15T11:25:24.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.088+0000 7f14b6ffd700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b80b1e40 0x7f14b80ce580 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f14a8015000 tx=0x7f14a8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.085+0000 7fc44bfff700 1 -- 172.21.15.113:0/569269716 <== mon.1 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc458004ab0 con 0x7fc44c004ca0 2024-01-15T11:25:24.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.085+0000 7fc44bfff700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc45006b3a0 0x7fc45006d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.086+0000 7fc44bfff700 1 -- 172.21.15.113:0/569269716 <== mon.1 v2:172.21.15.141:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc45809c0c0 con 0x7fc44c004ca0 2024-01-15T11:25:24.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.087+0000 7fc462ffd700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc45006b3a0 0x7fc45006d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.088+0000 7fc462ffd700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc45006b3a0 0x7fc45006d860 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc4640b3560 tx=0x7fc454009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.089+0000 7f149f7fe700 1 -- 172.21.15.113:0/493499869 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8003c50 con 0x7f14b80b1e40 2024-01-15T11:25:24.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.089+0000 7f149f7fe700 1 -- 172.21.15.113:0/493499869 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a8020070 con 0x7f14b80b1e40 2024-01-15T11:25:24.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.089+0000 7f149f7fe700 1 -- 172.21.15.113:0/493499869 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8011630 con 0x7f14b80b1e40 2024-01-15T11:25:24.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.089+0000 7f14bd28b700 1 -- 172.21.15.113:0/493499869 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b80b40a0 con 0x7f14b80b1e40 2024-01-15T11:25:24.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.089+0000 7f14bd28b700 1 -- 172.21.15.113:0/493499869 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b80cec80 con 0x7f14b80b1e40 2024-01-15T11:25:24.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.091+0000 7f149f7fe700 1 -- 172.21.15.113:0/493499869 <== mon.2 v2:172.21.15.141:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f14a8022030 con 0x7f14b80b1e40 2024-01-15T11:25:24.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.092+0000 7f149f7fe700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f14a006b530 0x7f14a006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.092+0000 7f149f7fe700 1 -- 172.21.15.113:0/493499869 <== mon.2 v2:172.21.15.141:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14a808b1b0 con 0x7f14b80b1e40 2024-01-15T11:25:24.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.092+0000 7f14b67fc700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f14a006b530 0x7f14a006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.093+0000 7f14b67fc700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f14a006b530 0x7f14a006d9f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f14b80d1150 tx=0x7f14ac0091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.093+0000 7f14bd28b700 1 -- 172.21.15.113:0/493499869 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a40052f0 con 0x7f14b80b1e40 2024-01-15T11:25:24.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.093+0000 7fc46a114700 1 -- 172.21.15.113:0/569269716 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4640c5190 con 0x7fc44c004ca0 2024-01-15T11:25:24.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.098+0000 7f149f7fe700 1 -- 172.21.15.113:0/493499869 <== mon.2 v2:172.21.15.141:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14a8056090 con 0x7f14b80b1e40 2024-01-15T11:25:24.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.101+0000 7fc44bfff700 1 -- 172.21.15.113:0/569269716 <== mon.1 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4640c5190 con 0x7fc44c004ca0 2024-01-15T11:25:24.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.144+0000 7f7d22c56700 1 Processor -- start 2024-01-15T11:25:24.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.144+0000 7f7d22c56700 1 -- start start 2024-01-15T11:25:24.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.145+0000 7f7d22c56700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7d1c070fd0 0x7f7d1c165e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.145+0000 7f7d22c56700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7d1c1663d0 0x7f7d1c0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.145+0000 7f7d22c56700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1c072110 con 0x7f7d1c0cf440 2024-01-15T11:25:24.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.145+0000 7f7d22c56700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d1c166860 con 0x7f7d1c070fd0 2024-01-15T11:25:24.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.145+0000 7f7d22c56700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c071500 con 0x7f7d1c1663d0 2024-01-15T11:25:24.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.146+0000 7f7d13fff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7d1c070fd0 0x7f7d1c165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.146+0000 7f7d209f2700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f7d1c0cf440 0x7f7d1c06e8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:35342/0 (socket says 172.21.15.113:35342) 2024-01-15T11:25:24.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.146+0000 7f7d209f2700 1 -- 172.21.15.113:0/3751535125 learned_addr learned my addr 172.21.15.113:0/3751535125 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:24.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.146+0000 7f7d211f3700 1 --2- 172.21.15.113:0/3751535125 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7d1c1663d0 0x7f7d1c0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.146+0000 7f7d137fe700 1 -- 172.21.15.113:0/3751535125 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395629259 0 0) 0x7f7d1c072110 con 0x7f7d1c0cf440 2024-01-15T11:25:24.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.147+0000 7f7d137fe700 1 -- 172.21.15.113:0/3751535125 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d040035b0 con 0x7f7d1c0cf440 2024-01-15T11:25:24.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.147+0000 7f7d211f3700 1 -- 172.21.15.113:0/3751535125 >> v2:172.21.15.141:3300/0 conn(0x7f7d1c070fd0 msgr2=0x7f7d1c165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:24.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.147+0000 7f7d211f3700 1 --2- 172.21.15.113:0/3751535125 >> v2:172.21.15.141:3300/0 conn(0x7f7d1c070fd0 0x7f7d1c165e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.147+0000 7f7d211f3700 1 -- 172.21.15.113:0/3751535125 >> v1:172.21.15.141:6789/0 conn(0x7f7d1c0cf440 legacy=0x7f7d1c06e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.147+0000 7f7d211f3700 1 -- 172.21.15.113:0/3751535125 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c0717e0 con 0x7f7d1c1663d0 2024-01-15T11:25:24.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.147+0000 7f7d211f3700 1 --2- 172.21.15.113:0/3751535125 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7d1c1663d0 0x7f7d1c0b2a40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009d90 tx=0x7f7d1400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a5f09f88193ca81 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.147+0000 7f7d137fe700 1 -- 172.21.15.113:0/3751535125 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1400e070 con 0x7f7d1c1663d0 2024-01-15T11:25:24.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.147+0000 7f7d137fe700 1 -- 172.21.15.113:0/3751535125 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1400bb20 con 0x7f7d1c1663d0 2024-01-15T11:25:24.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.147+0000 7f7d137fe700 1 -- 172.21.15.113:0/3751535125 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d14012430 con 0x7f7d1c1663d0 2024-01-15T11:25:24.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.148+0000 7f7d22c56700 1 -- 172.21.15.113:0/3751535125 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7d1c1663d0 msgr2=0x7f7d1c0b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.148+0000 7f7d22c56700 1 --2- 172.21.15.113:0/3751535125 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7d1c1663d0 0x7f7d1c0b2a40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009d90 tx=0x7f7d1400b330 comp rx=0 tx=0).stop 2024-01-15T11:25:24.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.148+0000 7f7d22c56700 1 -- 172.21.15.113:0/3751535125 shutdown_connections 2024-01-15T11:25:24.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.148+0000 7f7d22c56700 1 --2- 172.21.15.113:0/3751535125 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7d1c1663d0 0x7f7d1c0b2a40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.148+0000 7f7d22c56700 1 --2- 172.21.15.113:0/3751535125 >> v2:172.21.15.141:3300/0 conn(0x7f7d1c070fd0 0x7f7d1c165e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.148+0000 7f7d22c56700 1 -- 172.21.15.113:0/3751535125 >> 172.21.15.113:0/3751535125 conn(0x7f7d1c0b1740 msgr2=0x7f7d1c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.149+0000 7f7d22c56700 1 -- 172.21.15.113:0/3751535125 shutdown_connections 2024-01-15T11:25:24.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.149+0000 7f7d22c56700 1 -- 172.21.15.113:0/3751535125 wait complete. 2024-01-15T11:25:24.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.150+0000 7f7d22c56700 1 Processor -- start 2024-01-15T11:25:24.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.151+0000 7f7d22c56700 1 -- start start 2024-01-15T11:25:24.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.152+0000 7f7d22c56700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7d1c070fd0 0x7f7d1c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.152+0000 7f7d22c56700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d1c0cf440 0x7f7d1c0affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.152+0000 7f7d22c56700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d1c1663d0 0x7f7d1c133310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.152+0000 7f7d22c56700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c160ed0 con 0x7f7d1c0cf440 2024-01-15T11:25:24.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.152+0000 7f7d22c56700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c161490 con 0x7f7d1c1663d0 2024-01-15T11:25:24.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.152+0000 7f7d22c56700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c0b06f0 con 0x7f7d1c070fd0 2024-01-15T11:25:24.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.152+0000 7f7d13fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d1c0cf440 0x7f7d1c0affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.152+0000 7f7d13fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d1c0cf440 0x7f7d1c0affe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45838/0 (socket says 172.21.15.113:45838) 2024-01-15T11:25:24.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.152+0000 7f7d13fff700 1 -- 172.21.15.113:0/3355168320 learned_addr learned my addr 172.21.15.113:0/3355168320 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:24.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.153+0000 7f7d13fff700 1 -- 172.21.15.113:0/3355168320 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7d1c070fd0 msgr2=0x7f7d1c15c600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T11:25:24.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.153+0000 7f7d13fff700 1 --2- 172.21.15.113:0/3355168320 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7d1c070fd0 0x7f7d1c15c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.153+0000 7f7d13fff700 1 -- 172.21.15.113:0/3355168320 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d1c1663d0 msgr2=0x7f7d1c133310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T11:25:24.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.153+0000 7f7d13fff700 1 --2- 172.21.15.113:0/3355168320 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d1c1663d0 0x7f7d1c133310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.153+0000 7f7d13fff700 1 -- 172.21.15.113:0/3355168320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d14009a70 con 0x7f7d1c0cf440 2024-01-15T11:25:24.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.153+0000 7f7d13fff700 1 --2- 172.21.15.113:0/3355168320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d1c0cf440 0x7f7d1c0affe0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800e980 tx=0x7f7d1800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.154+0000 7f7d117fa700 1 -- 172.21.15.113:0/3355168320 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1800c850 con 0x7f7d1c0cf440 2024-01-15T11:25:24.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.154+0000 7f7d117fa700 1 -- 172.21.15.113:0/3355168320 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1800c9b0 con 0x7f7d1c0cf440 2024-01-15T11:25:24.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.154+0000 7f7d22c56700 1 -- 172.21.15.113:0/3355168320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d1c148a20 con 0x7f7d1c0cf440 2024-01-15T11:25:24.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.154+0000 7f7d22c56700 1 -- 172.21.15.113:0/3355168320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d1c161120 con 0x7f7d1c0cf440 2024-01-15T11:25:24.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.155+0000 7f7d117fa700 1 -- 172.21.15.113:0/3355168320 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1800cd20 con 0x7f7d1c0cf440 2024-01-15T11:25:24.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.160+0000 7f7d22c56700 1 -- 172.21.15.113:0/3355168320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cfc0052f0 con 0x7f7d1c0cf440 2024-01-15T11:25:24.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.166+0000 7f7d117fa700 1 -- 172.21.15.113:0/3355168320 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7d18005a20 con 0x7f7d1c0cf440 2024-01-15T11:25:24.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.166+0000 7f7d117fa700 1 --2- 172.21.15.113:0/3355168320 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7d0806b350 0x7f7d0806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:24.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.166+0000 7f7d117fa700 1 -- 172.21.15.113:0/3355168320 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d1808a510 con 0x7f7d1c0cf440 2024-01-15T11:25:24.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.170+0000 7f7d117fa700 1 -- 172.21.15.113:0/3355168320 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d180553a0 con 0x7f7d1c0cf440 2024-01-15T11:25:24.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.170+0000 7f7d209f2700 1 --2- 172.21.15.113:0/3355168320 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7d0806b350 0x7f7d0806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:24.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.171+0000 7f7d209f2700 1 --2- 172.21.15.113:0/3355168320 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7d0806b350 0x7f7d0806d810 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c007fd0 tx=0x7f7d0c007980 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:24.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.286+0000 7f688a4d6700 1 -- 172.21.15.113:0/1264768226 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f68840751e0 con 0x7f688415d610 2024-01-15T11:25:24.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.286+0000 7f6880ff9700 1 -- 172.21.15.113:0/1264768226 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f687c018020 con 0x7f688415d610 2024-01-15T11:25:24.285 INFO:teuthology.orchestra.run.smithi113.stdout:55834574854 2024-01-15T11:25:24.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.290+0000 7f688a4d6700 1 -- 172.21.15.113:0/1264768226 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f687006b3a0 msgr2=0x7f687006d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.290+0000 7f688a4d6700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f687006b3a0 0x7f687006d860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f68840cf5a0 tx=0x7f687400b040 comp rx=0 tx=0).stop 2024-01-15T11:25:24.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.290+0000 7f688a4d6700 1 -- 172.21.15.113:0/1264768226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f688415d610 msgr2=0x7f6884152260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.290+0000 7f688a4d6700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f688415d610 0x7f6884152260 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f687c00c7e0 tx=0x7f687c00cba0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.290+0000 7f688a4d6700 1 -- 172.21.15.113:0/1264768226 shutdown_connections 2024-01-15T11:25:24.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.290+0000 7f688a4d6700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f687006b3a0 0x7f687006d860 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f68840cf5a0 tx=0x7f687400b040 comp rx=0 tx=0).stop 2024-01-15T11:25:24.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.290+0000 7f688a4d6700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f688415d610 0x7f6884152260 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.290+0000 7f688a4d6700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68840714d0 0x7f6884153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.290+0000 7f688a4d6700 1 --2- 172.21.15.113:0/1264768226 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68840d0520 0x7f688413e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.290+0000 7f688a4d6700 1 -- 172.21.15.113:0/1264768226 >> 172.21.15.113:0/1264768226 conn(0x7f68840cfa30 msgr2=0x7f68840b0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.291+0000 7f688a4d6700 1 -- 172.21.15.113:0/1264768226 shutdown_connections 2024-01-15T11:25:24.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.291+0000 7f688a4d6700 1 -- 172.21.15.113:0/1264768226 wait complete. 2024-01-15T11:25:24.302 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.2 2024-01-15T11:25:24.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.394+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/503598552 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa4ac067f60 con 0x7fa4ac162240 2024-01-15T11:25:24.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.395+0000 7fa4a27fc700 1 -- 172.21.15.113:0/503598552 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa4a40576a0 con 0x7fa4ac162240 2024-01-15T11:25:24.394 INFO:teuthology.orchestra.run.smithi113.stdout:55834574855 2024-01-15T11:25:24.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.397+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/503598552 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa49006b420 msgr2=0x7fa49006d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.397+0000 7fa4b3fa2700 1 --2- 172.21.15.113:0/503598552 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa49006b420 0x7fa49006d8e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009670 tx=0x7fa4a8006bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.397+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/503598552 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4ac162240 msgr2=0x7fa4ac074200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.397+0000 7fa4b3fa2700 1 --2- 172.21.15.113:0/503598552 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4ac162240 0x7fa4ac074200 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400c780 tx=0x7fa4a400ca90 comp rx=0 tx=0).stop 2024-01-15T11:25:24.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.397+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/503598552 shutdown_connections 2024-01-15T11:25:24.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.397+0000 7fa4b3fa2700 1 --2- 172.21.15.113:0/503598552 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa49006b420 0x7fa49006d8e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.397+0000 7fa4b3fa2700 1 --2- 172.21.15.113:0/503598552 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4ac162240 0x7fa4ac074200 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.397+0000 7fa4b3fa2700 1 --2- 172.21.15.113:0/503598552 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4ac06f360 0x7fa4ac162920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.397+0000 7fa4b3fa2700 1 --2- 172.21.15.113:0/503598552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4ac0ae4e0 0x7fa4ac161580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.397+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/503598552 >> 172.21.15.113:0/503598552 conn(0x7fa4ac071b90 msgr2=0x7fa4ac13aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.397+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/503598552 shutdown_connections 2024-01-15T11:25:24.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.398+0000 7fa4b3fa2700 1 -- 172.21.15.113:0/503598552 wait complete. 2024-01-15T11:25:24.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.405+0000 7f2c00186700 1 -- 172.21.15.113:0/1142688803 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2bf8161290 con 0x7f2bf80d0d00 2024-01-15T11:25:24.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.405+0000 7f2bee7fc700 1 -- 172.21.15.113:0/1142688803 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2bf0054a40 con 0x7f2bf80d0d00 2024-01-15T11:25:24.404 INFO:teuthology.orchestra.run.smithi113.stdout:55834574857 2024-01-15T11:25:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.409+0000 7f2c00186700 1 -- 172.21.15.113:0/1142688803 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2bdc06b350 msgr2=0x7f2bdc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.409+0000 7f2c00186700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2bdc06b350 0x7f2bdc06d810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2be8007fd0 tx=0x7f2be8007dc0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.409+0000 7f2c00186700 1 -- 172.21.15.113:0/1142688803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bf80d0d00 msgr2=0x7f2bf815ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.409+0000 7f2c00186700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bf80d0d00 0x7f2bf815ff30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000cbe0 tx=0x7f2bf000cef0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.409+0000 7f2c00186700 1 -- 172.21.15.113:0/1142688803 shutdown_connections 2024-01-15T11:25:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.409+0000 7f2c00186700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2bdc06b350 0x7f2bdc06d810 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.409+0000 7f2c00186700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bf815e1e0 0x7f2bf815b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.409+0000 7f2c00186700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bf80d0870 0x7f2bf81f8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.409+0000 7f2c00186700 1 --2- 172.21.15.113:0/1142688803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bf80d0d00 0x7f2bf815ff30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.409+0000 7f2c00186700 1 -- 172.21.15.113:0/1142688803 >> 172.21.15.113:0/1142688803 conn(0x7f2bf80b5b50 msgr2=0x7f2bf80b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.409+0000 7f2c00186700 1 -- 172.21.15.113:0/1142688803 shutdown_connections 2024-01-15T11:25:24.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.409+0000 7f2c00186700 1 -- 172.21.15.113:0/1142688803 wait complete. 2024-01-15T11:25:24.409 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.0 2024-01-15T11:25:24.410 DEBUG:teuthology.parallel:result is None 2024-01-15T11:25:24.425 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-01-15T11:25:24.425 DEBUG:teuthology.parallel:result is None 2024-01-15T11:25:24.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.462+0000 7fc3674d0700 1 -- 172.21.15.113:0/1262110825 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc3400055a0 con 0x7fc36009e5d0 2024-01-15T11:25:24.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.462+0000 7fc355ffb700 1 -- 172.21.15.113:0/1262110825 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc358058380 con 0x7fc36009e5d0 2024-01-15T11:25:24.461 INFO:teuthology.orchestra.run.smithi113.stdout:55834574855 2024-01-15T11:25:24.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.463+0000 7f14bd28b700 1 -- 172.21.15.113:0/493499869 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f14a40055a0 con 0x7f14b80b1e40 2024-01-15T11:25:24.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.464+0000 7f149f7fe700 1 -- 172.21.15.113:0/493499869 <== mon.2 v2:172.21.15.141:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f14a80596b0 con 0x7f14b80b1e40 2024-01-15T11:25:24.462 INFO:teuthology.orchestra.run.smithi113.stdout:55834574856 2024-01-15T11:25:24.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.464+0000 7f7d22c56700 1 -- 172.21.15.113:0/3355168320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f7cfc005870 con 0x7f7d1c0cf440 2024-01-15T11:25:24.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.464+0000 7f7d117fa700 1 -- 172.21.15.113:0/3355168320 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7d18021020 con 0x7f7d1c0cf440 2024-01-15T11:25:24.463 INFO:teuthology.orchestra.run.smithi113.stdout:55834574854 2024-01-15T11:25:24.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7fc3674d0700 1 -- 172.21.15.113:0/1262110825 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc34c06b5b0 msgr2=0x7fc34c06da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7fc3674d0700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc34c06b5b0 0x7fc34c06da70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc360149fd0 tx=0x7fc35c010000 comp rx=0 tx=0).stop 2024-01-15T11:25:24.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7fc3674d0700 1 -- 172.21.15.113:0/1262110825 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc36009e5d0 msgr2=0x7fc3600cf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7fc3674d0700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc36009e5d0 0x7fc3600cf9e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc35800b780 tx=0x7fc35800bb40 comp rx=0 tx=0).stop 2024-01-15T11:25:24.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7f14bd28b700 1 -- 172.21.15.113:0/493499869 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f14a006b530 msgr2=0x7f14a006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7f14bd28b700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f14a006b530 0x7f14a006d9f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f14b80d1150 tx=0x7f14ac0091c0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7f14bd28b700 1 -- 172.21.15.113:0/493499869 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b80b1e40 msgr2=0x7f14b80ce580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7f14bd28b700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b80b1e40 0x7f14b80ce580 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f14a8015000 tx=0x7f14a8004750 comp rx=0 tx=0).stop 2024-01-15T11:25:24.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7f14bd28b700 1 -- 172.21.15.113:0/493499869 shutdown_connections 2024-01-15T11:25:24.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7f14bd28b700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f14a006b530 0x7f14a006d9f0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f14b80d1150 tx=0x7f14ac0091c0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7f14bd28b700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14b8163860 0x7f14b8148190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7f14bd28b700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f14b80b1e40 0x7f14b80ce580 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f14a8015000 tx=0x7f14a8004750 comp rx=0 tx=0).stop 2024-01-15T11:25:24.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7f14bd28b700 1 --2- 172.21.15.113:0/493499869 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f14b8163480 0x7f14b80d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7f14bd28b700 1 -- 172.21.15.113:0/493499869 >> 172.21.15.113:0/493499869 conn(0x7f14b80b57b0 msgr2=0x7f14b81619b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7f14bd28b700 1 -- 172.21.15.113:0/493499869 shutdown_connections 2024-01-15T11:25:24.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7f14bd28b700 1 -- 172.21.15.113:0/493499869 wait complete. 2024-01-15T11:25:24.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7fc3674d0700 1 -- 172.21.15.113:0/1262110825 shutdown_connections 2024-01-15T11:25:24.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7fc3674d0700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc34c06b5b0 0x7fc34c06da70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7fc3674d0700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc36009e5d0 0x7fc3600cf9e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7fc3674d0700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc36006c710 0x7fc3600cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7fc3674d0700 1 --2- 172.21.15.113:0/1262110825 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc360162240 0x7fc36014a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7fc3674d0700 1 -- 172.21.15.113:0/1262110825 >> 172.21.15.113:0/1262110825 conn(0x7fc360071b90 msgr2=0x7fc3600b16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7fc3674d0700 1 -- 172.21.15.113:0/1262110825 shutdown_connections 2024-01-15T11:25:24.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.466+0000 7fc3674d0700 1 -- 172.21.15.113:0/1262110825 wait complete. 2024-01-15T11:25:24.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.467+0000 7f7d22c56700 1 -- 172.21.15.113:0/3355168320 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7d0806b350 msgr2=0x7f7d0806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.467+0000 7f7d22c56700 1 --2- 172.21.15.113:0/3355168320 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7d0806b350 0x7f7d0806d810 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c007fd0 tx=0x7f7d0c007980 comp rx=0 tx=0).stop 2024-01-15T11:25:24.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.467+0000 7f7d22c56700 1 -- 172.21.15.113:0/3355168320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d1c0cf440 msgr2=0x7f7d1c0affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.467+0000 7f7d22c56700 1 --2- 172.21.15.113:0/3355168320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d1c0cf440 0x7f7d1c0affe0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800e980 tx=0x7f7d1800ed40 comp rx=0 tx=0).stop 2024-01-15T11:25:24.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.467+0000 7f7d22c56700 1 -- 172.21.15.113:0/3355168320 shutdown_connections 2024-01-15T11:25:24.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.467+0000 7f7d22c56700 1 --2- 172.21.15.113:0/3355168320 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7d0806b350 0x7f7d0806d810 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.467+0000 7f7d22c56700 1 --2- 172.21.15.113:0/3355168320 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7d1c1663d0 0x7f7d1c133310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.467+0000 7f7d22c56700 1 --2- 172.21.15.113:0/3355168320 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7d1c070fd0 0x7f7d1c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.467+0000 7f7d22c56700 1 --2- 172.21.15.113:0/3355168320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d1c0cf440 0x7f7d1c0affe0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.467+0000 7f7d22c56700 1 -- 172.21.15.113:0/3355168320 >> 172.21.15.113:0/3355168320 conn(0x7f7d1c0b1740 msgr2=0x7f7d1c14ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.467+0000 7f7d22c56700 1 -- 172.21.15.113:0/3355168320 shutdown_connections 2024-01-15T11:25:24.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.467+0000 7f7d22c56700 1 -- 172.21.15.113:0/3355168320 wait complete. 2024-01-15T11:25:24.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2024-01-15T11:25:24.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.482+0000 7fc46a114700 1 -- 172.21.15.113:0/569269716 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc4640ab530 con 0x7fc44c004ca0 2024-01-15T11:25:24.482 DEBUG:teuthology.parallel:result is None 2024-01-15T11:25:24.482 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2024-01-15T11:25:24.483 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2024-01-15T11:25:24.483 DEBUG:teuthology.parallel:result is None 2024-01-15T11:25:24.483 DEBUG:teuthology.parallel:result is None 2024-01-15T11:25:24.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.483+0000 7fc44bfff700 1 -- 172.21.15.113:0/569269716 <== mon.1 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc4640ab530 con 0x7fc44c004ca0 2024-01-15T11:25:24.483 INFO:teuthology.orchestra.run.smithi113.stdout:55834574856 2024-01-15T11:25:24.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.485+0000 7fc46a114700 1 -- 172.21.15.113:0/569269716 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc45006b3a0 msgr2=0x7fc45006d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.485+0000 7fc46a114700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc45006b3a0 0x7fc45006d860 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc4640b3560 tx=0x7fc454009040 comp rx=0 tx=0).stop 2024-01-15T11:25:24.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.485+0000 7fc46a114700 1 -- 172.21.15.113:0/569269716 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 msgr2=0x7fc4640ac720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:24.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.485+0000 7fc46a114700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 0x7fc4640ac720 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00c390 tx=0x7fc458001930 comp rx=0 tx=0).stop 2024-01-15T11:25:24.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.485+0000 7fc46a114700 1 -- 172.21.15.113:0/569269716 shutdown_connections 2024-01-15T11:25:24.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.485+0000 7fc46a114700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc45006b3a0 0x7fc45006d860 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.485+0000 7fc46a114700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4640ab150 0x7fc464156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.485+0000 7fc46a114700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc44c004ca0 0x7fc4640ac720 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.485+0000 7fc46a114700 1 --2- 172.21.15.113:0/569269716 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc4640cf1b0 0x7fc4640aac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:24.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.485+0000 7fc46a114700 1 -- 172.21.15.113:0/569269716 >> 172.21.15.113:0/569269716 conn(0x7fc4640cfa30 msgr2=0x7fc46406ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:24.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.485+0000 7fc46a114700 1 -- 172.21.15.113:0/569269716 shutdown_connections 2024-01-15T11:25:24.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:24.485+0000 7fc46a114700 1 -- 172.21.15.113:0/569269716 wait complete. 2024-01-15T11:25:24.496 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2024-01-15T11:25:24.496 DEBUG:teuthology.parallel:result is None 2024-01-15T11:25:25.303 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-15T11:25:25.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.432+0000 7f466d637700 1 Processor -- start 2024-01-15T11:25:25.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.432+0000 7f466d637700 1 -- start start 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.432+0000 7f466d637700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46680cfd80 0x7f46680d1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.432+0000 7f466d637700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46680d2080 0x7f46680b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.432+0000 7f466d637700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46680d0ee0 con 0x7f466815e090 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.432+0000 7f466d637700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46680d2510 con 0x7f46680cfd80 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.432+0000 7f466d637700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46680d02d0 con 0x7f46680d2080 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f46667fc700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46680cfd80 0x7f46680d1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f46677fe700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46680d2080 0x7f46680b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f46677fe700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46680d2080 0x7f46680b15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:35912/0 (socket says 172.21.15.113:35912) 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f4666ffd700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f466815e090 0x7f466806bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:41982/0 (socket says 172.21.15.113:41982) 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f46667fc700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46680cfd80 0x7f46680d1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:58228/0 (socket says 172.21.15.113:58228) 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f46677fe700 1 -- 172.21.15.113:0/2688345129 learned_addr learned my addr 172.21.15.113:0/2688345129 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f46677fe700 1 -- 172.21.15.113:0/2688345129 >> v2:172.21.15.141:3300/0 conn(0x7f46680cfd80 msgr2=0x7f46680d1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:25.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f46677fe700 1 --2- 172.21.15.113:0/2688345129 >> v2:172.21.15.141:3300/0 conn(0x7f46680cfd80 0x7f46680d1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f46677fe700 1 -- 172.21.15.113:0/2688345129 >> v1:172.21.15.141:6789/0 conn(0x7f466815e090 legacy=0x7f466806bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f46677fe700 1 -- 172.21.15.113:0/2688345129 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46680d05b0 con 0x7f46680d2080 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f46667fc700 1 --2- 172.21.15.113:0/2688345129 >> v2:172.21.15.141:3300/0 conn(0x7f46680cfd80 0x7f46680d1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f46677fe700 1 --2- 172.21.15.113:0/2688345129 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46680d2080 0x7f46680b15c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4660009e50 tx=0x7f466000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3355a144a8476341 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f4665ffb700 1 -- 172.21.15.113:0/2688345129 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466000e070 con 0x7f46680d2080 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f4665ffb700 1 -- 172.21.15.113:0/2688345129 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466000b910 con 0x7f46680d2080 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.433+0000 7f4665ffb700 1 -- 172.21.15.113:0/2688345129 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466000bcc0 con 0x7f46680d2080 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.434+0000 7f466d637700 1 -- 172.21.15.113:0/2688345129 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46680d2080 msgr2=0x7f46680b15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.434+0000 7f466d637700 1 --2- 172.21.15.113:0/2688345129 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46680d2080 0x7f46680b15c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4660009e50 tx=0x7f466000b330 comp rx=0 tx=0).stop 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.434+0000 7f466d637700 1 -- 172.21.15.113:0/2688345129 shutdown_connections 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.434+0000 7f466d637700 1 --2- 172.21.15.113:0/2688345129 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46680d2080 0x7f46680b15c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.434+0000 7f466d637700 1 --2- 172.21.15.113:0/2688345129 >> v2:172.21.15.141:3300/0 conn(0x7f46680cfd80 0x7f46680d1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.434+0000 7f466d637700 1 -- 172.21.15.113:0/2688345129 >> 172.21.15.113:0/2688345129 conn(0x7f46680731a0 msgr2=0x7f466809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.434+0000 7f466d637700 1 -- 172.21.15.113:0/2688345129 shutdown_connections 2024-01-15T11:25:25.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.434+0000 7f466d637700 1 -- 172.21.15.113:0/2688345129 wait complete. 2024-01-15T11:25:25.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.435+0000 7f466d637700 1 Processor -- start 2024-01-15T11:25:25.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.435+0000 7f466d637700 1 -- start start 2024-01-15T11:25:25.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f466d637700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46680cfd80 0x7f46680b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f466d637700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46680d2080 0x7f46681572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f466d637700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f466815e090 0x7f46680b54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f466d637700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46680b5d10 con 0x7f46680cfd80 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f466d637700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46680b5ff0 con 0x7f46680d2080 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f466d637700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46680b62d0 con 0x7f466815e090 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f4666ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46680cfd80 0x7f46680b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f4666ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46680cfd80 0x7f46680b3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37830/0 (socket says 172.21.15.113:37830) 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f4666ffd700 1 -- 172.21.15.113:0/1746848977 learned_addr learned my addr 172.21.15.113:0/1746848977 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f46667fc700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46680d2080 0x7f46681572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f46677fe700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f466815e090 0x7f46680b54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f4666ffd700 1 -- 172.21.15.113:0/1746848977 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f466815e090 msgr2=0x7f46680b54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f4666ffd700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f466815e090 0x7f46680b54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f4666ffd700 1 -- 172.21.15.113:0/1746848977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46680d2080 msgr2=0x7f46681572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f4666ffd700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46680d2080 0x7f46681572e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.436+0000 7f4666ffd700 1 -- 172.21.15.113:0/1746848977 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c0094a0 con 0x7f46680cfd80 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.437+0000 7f4666ffd700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46680cfd80 0x7f46680b3440 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f465800c500 tx=0x7f465800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.437+0000 7f464f7fe700 1 -- 172.21.15.113:0/1746848977 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658005cb0 con 0x7f46680cfd80 2024-01-15T11:25:25.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.437+0000 7f464f7fe700 1 -- 172.21.15.113:0/1746848977 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4658005e10 con 0x7f46680cfd80 2024-01-15T11:25:25.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.437+0000 7f466d637700 1 -- 172.21.15.113:0/1746848977 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4660009b30 con 0x7f46680cfd80 2024-01-15T11:25:25.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.437+0000 7f464f7fe700 1 -- 172.21.15.113:0/1746848977 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46580072d0 con 0x7f46680cfd80 2024-01-15T11:25:25.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.437+0000 7f466d637700 1 -- 172.21.15.113:0/1746848977 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46680b5a00 con 0x7f46680cfd80 2024-01-15T11:25:25.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.438+0000 7f466d637700 1 -- 172.21.15.113:0/1746848977 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46680751e0 con 0x7f46680cfd80 2024-01-15T11:25:25.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.438+0000 7f464f7fe700 1 -- 172.21.15.113:0/1746848977 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4658007430 con 0x7f46680cfd80 2024-01-15T11:25:25.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.438+0000 7f464f7fe700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f465406b350 0x7f465406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.438+0000 7f464f7fe700 1 -- 172.21.15.113:0/1746848977 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4658088f60 con 0x7f46680cfd80 2024-01-15T11:25:25.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.438+0000 7f46667fc700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f465406b350 0x7f465406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:25.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.439+0000 7f46667fc700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f465406b350 0x7f465406d810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f465c009c20 tx=0x7f465c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:25.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.443+0000 7f464f7fe700 1 -- 172.21.15.113:0/1746848977 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4658053df0 con 0x7f46680cfd80 2024-01-15T11:25:25.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.633+0000 7f466d637700 1 -- 172.21.15.113:0/1746848977 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f466815e960 con 0x7f46680cfd80 2024-01-15T11:25:25.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.633+0000 7f464f7fe700 1 -- 172.21.15.113:0/1746848977 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4658017070 con 0x7f46680cfd80 2024-01-15T11:25:25.632 INFO:teuthology.orchestra.run.smithi113.stdout:55834574856 2024-01-15T11:25:25.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.635+0000 7f466d637700 1 -- 172.21.15.113:0/1746848977 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f465406b350 msgr2=0x7f465406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:25.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.635+0000 7f466d637700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f465406b350 0x7f465406d810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f465c009c20 tx=0x7f465c008040 comp rx=0 tx=0).stop 2024-01-15T11:25:25.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.635+0000 7f466d637700 1 -- 172.21.15.113:0/1746848977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46680cfd80 msgr2=0x7f46680b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:25.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.635+0000 7f466d637700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46680cfd80 0x7f46680b3440 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f465800c500 tx=0x7f465800c8c0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.635+0000 7f466d637700 1 -- 172.21.15.113:0/1746848977 shutdown_connections 2024-01-15T11:25:25.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.635+0000 7f466d637700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f465406b350 0x7f465406d810 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.635+0000 7f466d637700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f466815e090 0x7f46680b54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.635+0000 7f466d637700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46680cfd80 0x7f46680b3440 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.635+0000 7f466d637700 1 --2- 172.21.15.113:0/1746848977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46680d2080 0x7f46681572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.635+0000 7f466d637700 1 -- 172.21.15.113:0/1746848977 >> 172.21.15.113:0/1746848977 conn(0x7f46680731a0 msgr2=0x7f46680c8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:25.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.635+0000 7f466d637700 1 -- 172.21.15.113:0/1746848977 shutdown_connections 2024-01-15T11:25:25.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.635+0000 7f466d637700 1 -- 172.21.15.113:0/1746848977 wait complete. 2024-01-15T11:25:25.645 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.2 2024-01-15T11:25:25.646 DEBUG:teuthology.parallel:result is None 2024-01-15T11:25:25.646 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-15T11:25:25.646 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T11:25:25.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.768+0000 7fafbd5af700 1 Processor -- start 2024-01-15T11:25:25.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.768+0000 7fafbd5af700 1 -- start start 2024-01-15T11:25:25.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafbd5af700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fafb815e1e0 0x7fafb80d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafbd5af700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fafb80d3a00 0x7fafb80d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafbd5af700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fafb80d0db0 0x7fafb806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafbd5af700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb80d3150 con 0x7fafb815e1e0 2024-01-15T11:25:25.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafbd5af700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb80d1240 con 0x7fafb80d3a00 2024-01-15T11:25:25.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafbd5af700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb815e5c0 con 0x7fafb80d0db0 2024-01-15T11:25:25.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb6ffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fafb815e1e0 0x7fafb80d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:25.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb6ffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fafb815e1e0 0x7fafb80d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37840/0 (socket says 172.21.15.113:37840) 2024-01-15T11:25:25.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb6ffd700 1 -- 172.21.15.113:0/2593357793 learned_addr learned my addr 172.21.15.113:0/2593357793 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb77fe700 1 --2- 172.21.15.113:0/2593357793 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fafb80d0db0 0x7fafb806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb67fc700 1 --2- 172.21.15.113:0/2593357793 >> v2:172.21.15.141:3300/0 conn(0x7fafb80d3a00 0x7fafb80d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb6ffd700 1 -- 172.21.15.113:0/2593357793 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fafb80d0db0 msgr2=0x7fafb806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb6ffd700 1 --2- 172.21.15.113:0/2593357793 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fafb80d0db0 0x7fafb806aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb6ffd700 1 -- 172.21.15.113:0/2593357793 >> v2:172.21.15.141:3300/0 conn(0x7fafb80d3a00 msgr2=0x7fafb80d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb6ffd700 1 --2- 172.21.15.113:0/2593357793 >> v2:172.21.15.141:3300/0 conn(0x7fafb80d3a00 0x7fafb80d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb6ffd700 1 -- 172.21.15.113:0/2593357793 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb80d20b0 con 0x7fafb815e1e0 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb6ffd700 1 --2- 172.21.15.113:0/2593357793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafb815e1e0 0x7fafb80d34c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fafb8161ba0 tx=0x7fafac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c403ef70acbfd9f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb5ffb700 1 -- 172.21.15.113:0/2593357793 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac00b860 con 0x7fafb815e1e0 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.769+0000 7fafb5ffb700 1 -- 172.21.15.113:0/2593357793 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac00b9c0 con 0x7fafb815e1e0 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.770+0000 7fafb5ffb700 1 -- 172.21.15.113:0/2593357793 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac00bd50 con 0x7fafb815e1e0 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.770+0000 7fafbd5af700 1 -- 172.21.15.113:0/2593357793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafb815e1e0 msgr2=0x7fafb80d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.770+0000 7fafbd5af700 1 --2- 172.21.15.113:0/2593357793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafb815e1e0 0x7fafb80d34c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fafb8161ba0 tx=0x7fafac009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.770+0000 7fafbd5af700 1 -- 172.21.15.113:0/2593357793 shutdown_connections 2024-01-15T11:25:25.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.770+0000 7fafbd5af700 1 --2- 172.21.15.113:0/2593357793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafb815e1e0 0x7fafb80d34c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.770+0000 7fafbd5af700 1 --2- 172.21.15.113:0/2593357793 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fafb80d0db0 0x7fafb806aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.770+0000 7fafbd5af700 1 --2- 172.21.15.113:0/2593357793 >> v2:172.21.15.141:3300/0 conn(0x7fafb80d3a00 0x7fafb80d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.770+0000 7fafbd5af700 1 -- 172.21.15.113:0/2593357793 >> 172.21.15.113:0/2593357793 conn(0x7fafb80b5b50 msgr2=0x7fafb80b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:25.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.770+0000 7fafbd5af700 1 -- 172.21.15.113:0/2593357793 shutdown_connections 2024-01-15T11:25:25.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.770+0000 7fafbd5af700 1 -- 172.21.15.113:0/2593357793 wait complete. 2024-01-15T11:25:25.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.771+0000 7fafbd5af700 1 Processor -- start 2024-01-15T11:25:25.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.771+0000 7fafbd5af700 1 -- start start 2024-01-15T11:25:25.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafbd5af700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fafb80d0db0 0x7fafb80ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafbd5af700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafb80d3a00 0x7fafb80ad380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafbd5af700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fafb815e1e0 0x7fafb80ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafbd5af700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb814d550 con 0x7fafb80d3a00 2024-01-15T11:25:25.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafbd5af700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb814cc20 con 0x7fafb80d0db0 2024-01-15T11:25:25.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafbd5af700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb814cf00 con 0x7fafb815e1e0 2024-01-15T11:25:25.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafb67fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafb80d3a00 0x7fafb80ad380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:25.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafb67fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafb80d3a00 0x7fafb80ad380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37844/0 (socket says 172.21.15.113:37844) 2024-01-15T11:25:25.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafb67fc700 1 -- 172.21.15.113:0/295258794 learned_addr learned my addr 172.21.15.113:0/295258794 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafb67fc700 1 -- 172.21.15.113:0/295258794 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fafb815e1e0 msgr2=0x7fafb80ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafb67fc700 1 --2- 172.21.15.113:0/295258794 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fafb815e1e0 0x7fafb80ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafb67fc700 1 -- 172.21.15.113:0/295258794 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fafb80d0db0 msgr2=0x7fafb80ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafb67fc700 1 --2- 172.21.15.113:0/295258794 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fafb80d0db0 0x7fafb80ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.772+0000 7fafb67fc700 1 -- 172.21.15.113:0/295258794 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac009550 con 0x7fafb80d3a00 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.773+0000 7fafb67fc700 1 --2- 172.21.15.113:0/295258794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafb80d3a00 0x7fafb80ad380 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fafa800e810 tx=0x7fafa800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.773+0000 7faf9f7fe700 1 -- 172.21.15.113:0/295258794 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8013610 con 0x7fafb80d3a00 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.773+0000 7faf9f7fe700 1 -- 172.21.15.113:0/295258794 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa8013770 con 0x7fafb80d3a00 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.773+0000 7fafbd5af700 1 -- 172.21.15.113:0/295258794 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb814d1e0 con 0x7fafb80d3a00 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.773+0000 7faf9f7fe700 1 -- 172.21.15.113:0/295258794 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8013b00 con 0x7fafb80d3a00 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.773+0000 7fafbd5af700 1 -- 172.21.15.113:0/295258794 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb80aa9d0 con 0x7fafb80d3a00 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.774+0000 7fafbd5af700 1 -- 172.21.15.113:0/295258794 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb814d370 con 0x7fafb80d3a00 2024-01-15T11:25:25.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.774+0000 7faf9f7fe700 1 -- 172.21.15.113:0/295258794 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fafa8013c60 con 0x7fafb80d3a00 2024-01-15T11:25:25.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.775+0000 7faf9f7fe700 1 --2- 172.21.15.113:0/295258794 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fafa406b350 0x7fafa406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:25.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.775+0000 7fafb6ffd700 1 --2- 172.21.15.113:0/295258794 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fafa406b350 0x7fafa406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:25.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.775+0000 7faf9f7fe700 1 -- 172.21.15.113:0/295258794 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafa80897e0 con 0x7fafb80d3a00 2024-01-15T11:25:25.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.775+0000 7fafb6ffd700 1 --2- 172.21.15.113:0/295258794 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fafa406b350 0x7fafa406d810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fafac014040 tx=0x7fafac010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:25.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.779+0000 7faf9f7fe700 1 -- 172.21.15.113:0/295258794 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fafa8054650 con 0x7fafb80d3a00 2024-01-15T11:25:25.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.967+0000 7fafbd5af700 1 -- 172.21.15.113:0/295258794 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fafb80aacb0 con 0x7fafa406b350 2024-01-15T11:25:25.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.974+0000 7faf9f7fe700 1 -- 172.21.15.113:0/295258794 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264606 (secure 0 0 0) 0x7fafb80aacb0 con 0x7fafa406b350 2024-01-15T11:25:25.976 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:25.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.977+0000 7fafbd5af700 1 -- 172.21.15.113:0/295258794 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fafa406b350 msgr2=0x7fafa406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:25.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.977+0000 7fafbd5af700 1 --2- 172.21.15.113:0/295258794 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fafa406b350 0x7fafa406d810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fafac014040 tx=0x7fafac010000 comp rx=0 tx=0).stop 2024-01-15T11:25:25.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.977+0000 7fafbd5af700 1 -- 172.21.15.113:0/295258794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafb80d3a00 msgr2=0x7fafb80ad380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:25.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.977+0000 7fafbd5af700 1 --2- 172.21.15.113:0/295258794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafb80d3a00 0x7fafb80ad380 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fafa800e810 tx=0x7fafa800ebd0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.977+0000 7fafbd5af700 1 -- 172.21.15.113:0/295258794 shutdown_connections 2024-01-15T11:25:25.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.977+0000 7fafbd5af700 1 --2- 172.21.15.113:0/295258794 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fafa406b350 0x7fafa406d810 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.977+0000 7fafbd5af700 1 --2- 172.21.15.113:0/295258794 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fafb815e1e0 0x7fafb80ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.977+0000 7fafbd5af700 1 --2- 172.21.15.113:0/295258794 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fafb80d0db0 0x7fafb80ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.977+0000 7fafbd5af700 1 --2- 172.21.15.113:0/295258794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafb80d3a00 0x7fafb80ad380 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:25.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.977+0000 7fafbd5af700 1 -- 172.21.15.113:0/295258794 >> 172.21.15.113:0/295258794 conn(0x7fafb80b5b50 msgr2=0x7fafb80ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:25.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.977+0000 7fafbd5af700 1 -- 172.21.15.113:0/295258794 shutdown_connections 2024-01-15T11:25:25.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:25.977+0000 7fafbd5af700 1 -- 172.21.15.113:0/295258794 wait complete. 2024-01-15T11:25:25.979 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-15T11:25:25.988 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-15T11:25:24.110561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219008,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755712,"statfs":{"total":773094113280,"available":772869849088,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001926"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.262398+0000","last_change":"2024-01-15T11:25:16.269790+0000","last_active":"2024-01-15T11:25:17.262398+0000","last_peered":"2024-01-15T11:25:17.262398+0000","last_clean":"2024-01-15T11:25:17.262398+0000","last_became_active":"2024-01-15T11:25:16.269540+0000","last_became_peered":"2024-01-15T11:25:16.269540+0000","last_unstale":"2024-01-15T11:25:17.262398+0000","last_undegraded":"2024-01-15T11:25:17.262398+0000","last_fullsized":"2024-01-15T11:25:17.262398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:47:05.456406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.256089+0000","last_change":"2024-01-15T11:25:16.254768+0000","last_active":"2024-01-15T11:25:17.256089+0000","last_peered":"2024-01-15T11:25:17.256089+0000","last_clean":"2024-01-15T11:25:17.256089+0000","last_became_active":"2024-01-15T11:25:16.254438+0000","last_became_peered":"2024-01-15T11:25:16.254438+0000","last_unstale":"2024-01-15T11:25:17.256089+0000","last_undegraded":"2024-01-15T11:25:17.256089+0000","last_fullsized":"2024-01-15T11:25:17.256089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:49.784954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836465+0000","last_change":"2024-01-15T11:25:16.255993+0000","last_active":"2024-01-15T11:25:18.836465+0000","last_peered":"2024-01-15T11:25:18.836465+0000","last_clean":"2024-01-15T11:25:18.836465+0000","last_became_active":"2024-01-15T11:25:16.255718+0000","last_became_peered":"2024-01-15T11:25:16.255718+0000","last_unstale":"2024-01-15T11:25:18.836465+0000","last_undegraded":"2024-01-15T11:25:18.836465+0000","last_fullsized":"2024-01-15T11:25:18.836465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:31.743954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870637+0000","last_change":"2024-01-15T11:25:16.262620+0000","last_active":"2024-01-15T11:25:18.870637+0000","last_peered":"2024-01-15T11:25:18.870637+0000","last_clean":"2024-01-15T11:25:18.870637+0000","last_became_active":"2024-01-15T11:25:16.262147+0000","last_became_peered":"2024-01-15T11:25:16.262147+0000","last_unstale":"2024-01-15T11:25:18.870637+0000","last_undegraded":"2024-01-15T11:25:18.870637+0000","last_fullsized":"2024-01-15T11:25:18.870637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.346395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260464+0000","last_change":"2024-01-15T11:25:16.252826+0000","last_active":"2024-01-15T11:25:17.260464+0000","last_peered":"2024-01-15T11:25:17.260464+0000","last_clean":"2024-01-15T11:25:17.260464+0000","last_became_active":"2024-01-15T11:25:16.251976+0000","last_became_peered":"2024-01-15T11:25:16.251976+0000","last_unstale":"2024-01-15T11:25:17.260464+0000","last_undegraded":"2024-01-15T11:25:17.260464+0000","last_fullsized":"2024-01-15T11:25:17.260464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:15:36.291852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870521+0000","last_change":"2024-01-15T11:25:16.266650+0000","last_active":"2024-01-15T11:25:18.870521+0000","last_peered":"2024-01-15T11:25:18.870521+0000","last_clean":"2024-01-15T11:25:18.870521+0000","last_became_active":"2024-01-15T11:25:16.266157+0000","last_became_peered":"2024-01-15T11:25:16.266157+0000","last_unstale":"2024-01-15T11:25:18.870521+0000","last_undegraded":"2024-01-15T11:25:18.870521+0000","last_fullsized":"2024-01-15T11:25:18.870521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:39:35.147679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260275+0000","last_change":"2024-01-15T11:25:16.255535+0000","last_active":"2024-01-15T11:25:17.260275+0000","last_peered":"2024-01-15T11:25:17.260275+0000","last_clean":"2024-01-15T11:25:17.260275+0000","last_became_active":"2024-01-15T11:25:16.255248+0000","last_became_peered":"2024-01-15T11:25:16.255248+0000","last_unstale":"2024-01-15T11:25:17.260275+0000","last_undegraded":"2024-01-15T11:25:17.260275+0000","last_fullsized":"2024-01-15T11:25:17.260275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.346395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.869947+0000","last_change":"2024-01-15T11:25:16.258135+0000","last_active":"2024-01-15T11:25:18.869947+0000","last_peered":"2024-01-15T11:25:18.869947+0000","last_clean":"2024-01-15T11:25:18.869947+0000","last_became_active":"2024-01-15T11:25:16.257859+0000","last_became_peered":"2024-01-15T11:25:16.257859+0000","last_unstale":"2024-01-15T11:25:18.869947+0000","last_undegraded":"2024-01-15T11:25:18.869947+0000","last_fullsized":"2024-01-15T11:25:18.869947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:56:33.010927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.262562+0000","last_change":"2024-01-15T11:25:16.261234+0000","last_active":"2024-01-15T11:25:17.262562+0000","last_peered":"2024-01-15T11:25:17.262562+0000","last_clean":"2024-01-15T11:25:17.262562+0000","last_became_active":"2024-01-15T11:25:16.260952+0000","last_became_peered":"2024-01-15T11:25:16.260952+0000","last_unstale":"2024-01-15T11:25:17.262562+0000","last_undegraded":"2024-01-15T11:25:17.262562+0000","last_fullsized":"2024-01-15T11:25:17.262562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:22:29.117384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.281188+0000","last_change":"2024-01-15T11:25:16.266369+0000","last_active":"2024-01-15T11:25:18.281188+0000","last_peered":"2024-01-15T11:25:18.281188+0000","last_clean":"2024-01-15T11:25:18.281188+0000","last_became_active":"2024-01-15T11:25:16.266091+0000","last_became_peered":"2024-01-15T11:25:16.266091+0000","last_unstale":"2024-01-15T11:25:18.281188+0000","last_undegraded":"2024-01-15T11:25:18.281188+0000","last_fullsized":"2024-01-15T11:25:18.281188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:28:18.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.281095+0000","last_change":"2024-01-15T11:25:16.259582+0000","last_active":"2024-01-15T11:25:18.281095+0000","last_peered":"2024-01-15T11:25:18.281095+0000","last_clean":"2024-01-15T11:25:18.281095+0000","last_became_active":"2024-01-15T11:25:16.259270+0000","last_became_peered":"2024-01-15T11:25:16.259270+0000","last_unstale":"2024-01-15T11:25:18.281095+0000","last_undegraded":"2024-01-15T11:25:18.281095+0000","last_fullsized":"2024-01-15T11:25:18.281095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:31.743954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.262196+0000","last_change":"2024-01-15T11:25:16.274052+0000","last_active":"2024-01-15T11:25:17.262196+0000","last_peered":"2024-01-15T11:25:17.262196+0000","last_clean":"2024-01-15T11:25:17.262196+0000","last_became_active":"2024-01-15T11:25:16.273798+0000","last_became_peered":"2024-01-15T11:25:16.273798+0000","last_unstale":"2024-01-15T11:25:17.262196+0000","last_undegraded":"2024-01-15T11:25:17.262196+0000","last_fullsized":"2024-01-15T11:25:17.262196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:05.228308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870217+0000","last_change":"2024-01-15T11:25:16.250458+0000","last_active":"2024-01-15T11:25:18.870217+0000","last_peered":"2024-01-15T11:25:18.870217+0000","last_clean":"2024-01-15T11:25:18.870217+0000","last_became_active":"2024-01-15T11:25:16.250262+0000","last_became_peered":"2024-01-15T11:25:16.250262+0000","last_unstale":"2024-01-15T11:25:18.870217+0000","last_undegraded":"2024-01-15T11:25:18.870217+0000","last_fullsized":"2024-01-15T11:25:18.870217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:31.743954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836352+0000","last_change":"2024-01-15T11:25:16.253950+0000","last_active":"2024-01-15T11:25:18.836352+0000","last_peered":"2024-01-15T11:25:18.836352+0000","last_clean":"2024-01-15T11:25:18.836352+0000","last_became_active":"2024-01-15T11:25:16.253667+0000","last_became_peered":"2024-01-15T11:25:16.253667+0000","last_unstale":"2024-01-15T11:25:18.836352+0000","last_undegraded":"2024-01-15T11:25:18.836352+0000","last_fullsized":"2024-01-15T11:25:18.836352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:56:33.010927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.255356+0000","last_change":"2024-01-15T11:25:16.268387+0000","last_active":"2024-01-15T11:25:17.255356+0000","last_peered":"2024-01-15T11:25:17.255356+0000","last_clean":"2024-01-15T11:25:17.255356+0000","last_became_active":"2024-01-15T11:25:16.268088+0000","last_became_peered":"2024-01-15T11:25:16.268088+0000","last_unstale":"2024-01-15T11:25:17.255356+0000","last_undegraded":"2024-01-15T11:25:17.255356+0000","last_fullsized":"2024-01-15T11:25:17.255356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:02:52.680406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870265+0000","last_change":"2024-01-15T11:25:16.261989+0000","last_active":"2024-01-15T11:25:18.870265+0000","last_peered":"2024-01-15T11:25:18.870265+0000","last_clean":"2024-01-15T11:25:18.870265+0000","last_became_active":"2024-01-15T11:25:16.261260+0000","last_became_peered":"2024-01-15T11:25:16.261260+0000","last_unstale":"2024-01-15T11:25:18.870265+0000","last_undegraded":"2024-01-15T11:25:18.870265+0000","last_fullsized":"2024-01-15T11:25:18.870265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:28:18.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870959+0000","last_change":"2024-01-15T11:25:16.270904+0000","last_active":"2024-01-15T11:25:18.870959+0000","last_peered":"2024-01-15T11:25:18.870959+0000","last_clean":"2024-01-15T11:25:18.870959+0000","last_became_active":"2024-01-15T11:25:16.270611+0000","last_became_peered":"2024-01-15T11:25:16.270611+0000","last_unstale":"2024-01-15T11:25:18.870959+0000","last_undegraded":"2024-01-15T11:25:18.870959+0000","last_fullsized":"2024-01-15T11:25:18.870959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:18:26.824601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870399+0000","last_change":"2024-01-15T11:25:16.257019+0000","last_active":"2024-01-15T11:25:18.870399+0000","last_peered":"2024-01-15T11:25:18.870399+0000","last_clean":"2024-01-15T11:25:18.870399+0000","last_became_active":"2024-01-15T11:25:16.256278+0000","last_became_peered":"2024-01-15T11:25:16.256278+0000","last_unstale":"2024-01-15T11:25:18.870399+0000","last_undegraded":"2024-01-15T11:25:18.870399+0000","last_fullsized":"2024-01-15T11:25:18.870399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:05.228308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260725+0000","last_change":"2024-01-15T11:25:16.248798+0000","last_active":"2024-01-15T11:25:17.260725+0000","last_peered":"2024-01-15T11:25:17.260725+0000","last_clean":"2024-01-15T11:25:17.260725+0000","last_became_active":"2024-01-15T11:25:16.248488+0000","last_became_peered":"2024-01-15T11:25:16.248488+0000","last_unstale":"2024-01-15T11:25:17.260725+0000","last_undegraded":"2024-01-15T11:25:17.260725+0000","last_fullsized":"2024-01-15T11:25:17.260725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:37:51.629738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.148831+0000","last_change":"2024-01-15T11:25:15.233384+0000","last_active":"2024-01-15T11:25:17.148831+0000","last_peered":"2024-01-15T11:25:17.148831+0000","last_clean":"2024-01-15T11:25:17.148831+0000","last_became_active":"2024-01-15T11:25:15.233134+0000","last_became_peered":"2024-01-15T11:25:15.233134+0000","last_unstale":"2024-01-15T11:25:17.148831+0000","last_undegraded":"2024-01-15T11:25:17.148831+0000","last_fullsized":"2024-01-15T11:25:17.148831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870182+0000","last_change":"2024-01-15T11:25:16.256579+0000","last_active":"2024-01-15T11:25:18.870182+0000","last_peered":"2024-01-15T11:25:18.870182+0000","last_clean":"2024-01-15T11:25:18.870182+0000","last_became_active":"2024-01-15T11:25:16.256320+0000","last_became_peered":"2024-01-15T11:25:16.256320+0000","last_unstale":"2024-01-15T11:25:18.870182+0000","last_undegraded":"2024-01-15T11:25:18.870182+0000","last_fullsized":"2024-01-15T11:25:18.870182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.346395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121397+0000","last_change":"2024-01-15T11:25:15.230417+0000","last_active":"2024-01-15T11:25:17.121397+0000","last_peered":"2024-01-15T11:25:17.121397+0000","last_clean":"2024-01-15T11:25:17.121397+0000","last_became_active":"2024-01-15T11:25:15.230135+0000","last_became_peered":"2024-01-15T11:25:15.230135+0000","last_unstale":"2024-01-15T11:25:17.121397+0000","last_undegraded":"2024-01-15T11:25:17.121397+0000","last_fullsized":"2024-01-15T11:25:17.121397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:25.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836423+0000","last_change":"2024-01-15T11:25:16.259985+0000","last_active":"2024-01-15T11:25:18.836423+0000","last_peered":"2024-01-15T11:25:18.836423+0000","last_clean":"2024-01-15T11:25:18.836423+0000","last_became_active":"2024-01-15T11:25:16.259697+0000","last_became_peered":"2024-01-15T11:25:16.259697+0000","last_unstale":"2024-01-15T11:25:18.836423+0000","last_undegraded":"2024-01-15T11:25:18.836423+0000","last_fullsized":"2024-01-15T11:25:18.836423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:17:27.965755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118258+0000","last_change":"2024-01-15T11:25:15.238695+0000","last_active":"2024-01-15T11:25:17.118258+0000","last_peered":"2024-01-15T11:25:17.118258+0000","last_clean":"2024-01-15T11:25:17.118258+0000","last_became_active":"2024-01-15T11:25:15.238377+0000","last_became_peered":"2024-01-15T11:25:15.238377+0000","last_unstale":"2024-01-15T11:25:17.118258+0000","last_undegraded":"2024-01-15T11:25:17.118258+0000","last_fullsized":"2024-01-15T11:25:17.118258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:48.780156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870868+0000","last_change":"2024-01-15T11:25:16.264379+0000","last_active":"2024-01-15T11:25:18.870868+0000","last_peered":"2024-01-15T11:25:18.870868+0000","last_clean":"2024-01-15T11:25:18.870868+0000","last_became_active":"2024-01-15T11:25:16.264121+0000","last_became_peered":"2024-01-15T11:25:16.264121+0000","last_unstale":"2024-01-15T11:25:18.870868+0000","last_undegraded":"2024-01-15T11:25:18.870868+0000","last_fullsized":"2024-01-15T11:25:18.870868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:37:51.629738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118194+0000","last_change":"2024-01-15T11:25:15.237423+0000","last_active":"2024-01-15T11:25:17.118194+0000","last_peered":"2024-01-15T11:25:17.118194+0000","last_clean":"2024-01-15T11:25:17.118194+0000","last_became_active":"2024-01-15T11:25:15.237121+0000","last_became_peered":"2024-01-15T11:25:15.237121+0000","last_unstale":"2024-01-15T11:25:17.118194+0000","last_undegraded":"2024-01-15T11:25:17.118194+0000","last_fullsized":"2024-01-15T11:25:17.118194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:04.223510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870892+0000","last_change":"2024-01-15T11:25:16.266842+0000","last_active":"2024-01-15T11:25:18.870892+0000","last_peered":"2024-01-15T11:25:18.870892+0000","last_clean":"2024-01-15T11:25:18.870892+0000","last_became_active":"2024-01-15T11:25:16.266606+0000","last_became_peered":"2024-01-15T11:25:16.266606+0000","last_unstale":"2024-01-15T11:25:18.870892+0000","last_undegraded":"2024-01-15T11:25:18.870892+0000","last_fullsized":"2024-01-15T11:25:18.870892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:37.335593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121353+0000","last_change":"2024-01-15T11:25:15.232263+0000","last_active":"2024-01-15T11:25:17.121353+0000","last_peered":"2024-01-15T11:25:17.121353+0000","last_clean":"2024-01-15T11:25:17.121353+0000","last_became_active":"2024-01-15T11:25:15.231976+0000","last_became_peered":"2024-01-15T11:25:15.231976+0000","last_unstale":"2024-01-15T11:25:17.121353+0000","last_undegraded":"2024-01-15T11:25:17.121353+0000","last_fullsized":"2024-01-15T11:25:17.121353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:25.797063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.261726+0000","last_change":"2024-01-15T11:25:16.263430+0000","last_active":"2024-01-15T11:25:17.261726+0000","last_peered":"2024-01-15T11:25:17.261726+0000","last_clean":"2024-01-15T11:25:17.261726+0000","last_became_active":"2024-01-15T11:25:16.261944+0000","last_became_peered":"2024-01-15T11:25:16.261944+0000","last_unstale":"2024-01-15T11:25:17.261726+0000","last_undegraded":"2024-01-15T11:25:17.261726+0000","last_fullsized":"2024-01-15T11:25:17.261726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.431909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.125258+0000","last_change":"2024-01-15T11:25:15.235451+0000","last_active":"2024-01-15T11:25:17.125258+0000","last_peered":"2024-01-15T11:25:17.125258+0000","last_clean":"2024-01-15T11:25:17.125258+0000","last_became_active":"2024-01-15T11:25:15.235166+0000","last_became_peered":"2024-01-15T11:25:15.235166+0000","last_unstale":"2024-01-15T11:25:17.125258+0000","last_undegraded":"2024-01-15T11:25:17.125258+0000","last_fullsized":"2024-01-15T11:25:17.125258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836642+0000","last_change":"2024-01-15T11:25:16.262932+0000","last_active":"2024-01-15T11:25:18.836642+0000","last_peered":"2024-01-15T11:25:18.836642+0000","last_clean":"2024-01-15T11:25:18.836642+0000","last_became_active":"2024-01-15T11:25:16.262667+0000","last_became_peered":"2024-01-15T11:25:16.262667+0000","last_unstale":"2024-01-15T11:25:18.836642+0000","last_undegraded":"2024-01-15T11:25:18.836642+0000","last_fullsized":"2024-01-15T11:25:18.836642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:49.784954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121325+0000","last_change":"2024-01-15T11:25:15.232286+0000","last_active":"2024-01-15T11:25:17.121325+0000","last_peered":"2024-01-15T11:25:17.121325+0000","last_clean":"2024-01-15T11:25:17.121325+0000","last_became_active":"2024-01-15T11:25:15.230764+0000","last_became_peered":"2024-01-15T11:25:15.230764+0000","last_unstale":"2024-01-15T11:25:17.121325+0000","last_undegraded":"2024-01-15T11:25:17.121325+0000","last_fullsized":"2024-01-15T11:25:17.121325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:53.755186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870515+0000","last_change":"2024-01-15T11:25:16.254427+0000","last_active":"2024-01-15T11:25:18.870515+0000","last_peered":"2024-01-15T11:25:18.870515+0000","last_clean":"2024-01-15T11:25:18.870515+0000","last_became_active":"2024-01-15T11:25:16.253271+0000","last_became_peered":"2024-01-15T11:25:16.253271+0000","last_unstale":"2024-01-15T11:25:18.870515+0000","last_undegraded":"2024-01-15T11:25:18.870515+0000","last_fullsized":"2024-01-15T11:25:18.870515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:08:06.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.128744+0000","last_change":"2024-01-15T11:25:15.231253+0000","last_active":"2024-01-15T11:25:17.128744+0000","last_peered":"2024-01-15T11:25:17.128744+0000","last_clean":"2024-01-15T11:25:17.128744+0000","last_became_active":"2024-01-15T11:25:15.230935+0000","last_became_peered":"2024-01-15T11:25:15.230935+0000","last_unstale":"2024-01-15T11:25:17.128744+0000","last_undegraded":"2024-01-15T11:25:17.128744+0000","last_fullsized":"2024-01-15T11:25:17.128744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:04.223510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.255483+0000","last_change":"2024-01-15T11:25:16.256798+0000","last_active":"2024-01-15T11:25:17.255483+0000","last_peered":"2024-01-15T11:25:17.255483+0000","last_clean":"2024-01-15T11:25:17.255483+0000","last_became_active":"2024-01-15T11:25:16.256540+0000","last_became_peered":"2024-01-15T11:25:16.256540+0000","last_unstale":"2024-01-15T11:25:17.255483+0000","last_undegraded":"2024-01-15T11:25:17.255483+0000","last_fullsized":"2024-01-15T11:25:17.255483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:15.719082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.125424+0000","last_change":"2024-01-15T11:25:15.236440+0000","last_active":"2024-01-15T11:25:17.125424+0000","last_peered":"2024-01-15T11:25:17.125424+0000","last_clean":"2024-01-15T11:25:17.125424+0000","last_became_active":"2024-01-15T11:25:15.236034+0000","last_became_peered":"2024-01-15T11:25:15.236034+0000","last_unstale":"2024-01-15T11:25:17.125424+0000","last_undegraded":"2024-01-15T11:25:17.125424+0000","last_fullsized":"2024-01-15T11:25:17.125424+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:25.797063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870721+0000","last_change":"2024-01-15T11:25:16.270509+0000","last_active":"2024-01-15T11:25:18.870721+0000","last_peered":"2024-01-15T11:25:18.870721+0000","last_clean":"2024-01-15T11:25:18.870721+0000","last_became_active":"2024-01-15T11:25:16.270319+0000","last_became_peered":"2024-01-15T11:25:16.270319+0000","last_unstale":"2024-01-15T11:25:18.870721+0000","last_undegraded":"2024-01-15T11:25:18.870721+0000","last_fullsized":"2024-01-15T11:25:18.870721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:15:36.291852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.128892+0000","last_change":"2024-01-15T11:25:15.235314+0000","last_active":"2024-01-15T11:25:17.128892+0000","last_peered":"2024-01-15T11:25:17.128892+0000","last_clean":"2024-01-15T11:25:17.128892+0000","last_became_active":"2024-01-15T11:25:15.232515+0000","last_became_peered":"2024-01-15T11:25:15.232515+0000","last_unstale":"2024-01-15T11:25:17.128892+0000","last_undegraded":"2024-01-15T11:25:17.128892+0000","last_fullsized":"2024-01-15T11:25:17.128892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:32.421454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836642+0000","last_change":"2024-01-15T11:25:16.260223+0000","last_active":"2024-01-15T11:25:18.836642+0000","last_peered":"2024-01-15T11:25:18.836642+0000","last_clean":"2024-01-15T11:25:18.836642+0000","last_became_active":"2024-01-15T11:25:16.259365+0000","last_became_peered":"2024-01-15T11:25:16.259365+0000","last_unstale":"2024-01-15T11:25:18.836642+0000","last_undegraded":"2024-01-15T11:25:18.836642+0000","last_fullsized":"2024-01-15T11:25:18.836642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:15:35.970176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.124986+0000","last_change":"2024-01-15T11:25:15.233433+0000","last_active":"2024-01-15T11:25:17.124986+0000","last_peered":"2024-01-15T11:25:17.124986+0000","last_clean":"2024-01-15T11:25:17.124986+0000","last_became_active":"2024-01-15T11:25:15.233046+0000","last_became_peered":"2024-01-15T11:25:15.233046+0000","last_unstale":"2024-01-15T11:25:17.124986+0000","last_undegraded":"2024-01-15T11:25:17.124986+0000","last_fullsized":"2024-01-15T11:25:17.124986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:25.797063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.262050+0000","last_change":"2024-01-15T11:25:16.275023+0000","last_active":"2024-01-15T11:25:17.262050+0000","last_peered":"2024-01-15T11:25:17.262050+0000","last_clean":"2024-01-15T11:25:17.262050+0000","last_became_active":"2024-01-15T11:25:16.274744+0000","last_became_peered":"2024-01-15T11:25:16.274744+0000","last_unstale":"2024-01-15T11:25:17.262050+0000","last_undegraded":"2024-01-15T11:25:17.262050+0000","last_fullsized":"2024-01-15T11:25:17.262050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.125282+0000","last_change":"2024-01-15T11:25:15.235460+0000","last_active":"2024-01-15T11:25:17.125282+0000","last_peered":"2024-01-15T11:25:17.125282+0000","last_clean":"2024-01-15T11:25:17.125282+0000","last_became_active":"2024-01-15T11:25:15.235194+0000","last_became_peered":"2024-01-15T11:25:15.235194+0000","last_unstale":"2024-01-15T11:25:17.125282+0000","last_undegraded":"2024-01-15T11:25:17.125282+0000","last_fullsized":"2024-01-15T11:25:17.125282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:32.421454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260798+0000","last_change":"2024-01-15T11:25:16.254349+0000","last_active":"2024-01-15T11:25:17.260798+0000","last_peered":"2024-01-15T11:25:17.260798+0000","last_clean":"2024-01-15T11:25:17.260798+0000","last_became_active":"2024-01-15T11:25:16.254068+0000","last_became_peered":"2024-01-15T11:25:16.254068+0000","last_unstale":"2024-01-15T11:25:17.260798+0000","last_undegraded":"2024-01-15T11:25:17.260798+0000","last_fullsized":"2024-01-15T11:25:17.260798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:37.335593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.148740+0000","last_change":"2024-01-15T11:25:15.236870+0000","last_active":"2024-01-15T11:25:17.148740+0000","last_peered":"2024-01-15T11:25:17.148740+0000","last_clean":"2024-01-15T11:25:17.148740+0000","last_became_active":"2024-01-15T11:25:15.236597+0000","last_became_peered":"2024-01-15T11:25:15.236597+0000","last_unstale":"2024-01-15T11:25:17.148740+0000","last_undegraded":"2024-01-15T11:25:17.148740+0000","last_fullsized":"2024-01-15T11:25:17.148740+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:04.223510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836341+0000","last_change":"2024-01-15T11:25:16.257709+0000","last_active":"2024-01-15T11:25:18.836341+0000","last_peered":"2024-01-15T11:25:18.836341+0000","last_clean":"2024-01-15T11:25:18.836341+0000","last_became_active":"2024-01-15T11:25:16.257415+0000","last_became_peered":"2024-01-15T11:25:16.257415+0000","last_unstale":"2024-01-15T11:25:18.836341+0000","last_undegraded":"2024-01-15T11:25:18.836341+0000","last_fullsized":"2024-01-15T11:25:18.836341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:26.801860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.128807+0000","last_change":"2024-01-15T11:25:15.233904+0000","last_active":"2024-01-15T11:25:17.128807+0000","last_peered":"2024-01-15T11:25:17.128807+0000","last_clean":"2024-01-15T11:25:17.128807+0000","last_became_active":"2024-01-15T11:25:15.232303+0000","last_became_peered":"2024-01-15T11:25:15.232303+0000","last_unstale":"2024-01-15T11:25:17.128807+0000","last_undegraded":"2024-01-15T11:25:17.128807+0000","last_fullsized":"2024-01-15T11:25:17.128807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:14.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260640+0000","last_change":"2024-01-15T11:25:16.265440+0000","last_active":"2024-01-15T11:25:17.260640+0000","last_peered":"2024-01-15T11:25:17.260640+0000","last_clean":"2024-01-15T11:25:17.260640+0000","last_became_active":"2024-01-15T11:25:16.265124+0000","last_became_peered":"2024-01-15T11:25:16.265124+0000","last_unstale":"2024-01-15T11:25:17.260640+0000","last_undegraded":"2024-01-15T11:25:17.260640+0000","last_fullsized":"2024-01-15T11:25:17.260640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:28:28.895688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.125216+0000","last_change":"2024-01-15T11:25:15.230668+0000","last_active":"2024-01-15T11:25:17.125216+0000","last_peered":"2024-01-15T11:25:17.125216+0000","last_clean":"2024-01-15T11:25:17.125216+0000","last_became_active":"2024-01-15T11:25:15.230323+0000","last_became_peered":"2024-01-15T11:25:15.230323+0000","last_unstale":"2024-01-15T11:25:17.125216+0000","last_undegraded":"2024-01-15T11:25:17.125216+0000","last_fullsized":"2024-01-15T11:25:17.125216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:25.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.280889+0000","last_change":"2024-01-15T11:25:16.267799+0000","last_active":"2024-01-15T11:25:18.280889+0000","last_peered":"2024-01-15T11:25:18.280889+0000","last_clean":"2024-01-15T11:25:18.280889+0000","last_became_active":"2024-01-15T11:25:16.266612+0000","last_became_peered":"2024-01-15T11:25:16.266612+0000","last_unstale":"2024-01-15T11:25:18.280889+0000","last_undegraded":"2024-01-15T11:25:18.280889+0000","last_fullsized":"2024-01-15T11:25:18.280889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:17:27.965755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121479+0000","last_change":"2024-01-15T11:25:15.236904+0000","last_active":"2024-01-15T11:25:17.121479+0000","last_peered":"2024-01-15T11:25:17.121479+0000","last_clean":"2024-01-15T11:25:17.121479+0000","last_became_active":"2024-01-15T11:25:15.236628+0000","last_became_peered":"2024-01-15T11:25:15.236628+0000","last_unstale":"2024-01-15T11:25:17.121479+0000","last_undegraded":"2024-01-15T11:25:17.121479+0000","last_fullsized":"2024-01-15T11:25:17.121479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:42.327446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260517+0000","last_change":"2024-01-15T11:25:16.271784+0000","last_active":"2024-01-15T11:25:17.260517+0000","last_peered":"2024-01-15T11:25:17.260517+0000","last_clean":"2024-01-15T11:25:17.260517+0000","last_became_active":"2024-01-15T11:25:16.271432+0000","last_became_peered":"2024-01-15T11:25:16.271432+0000","last_unstale":"2024-01-15T11:25:17.260517+0000","last_undegraded":"2024-01-15T11:25:17.260517+0000","last_fullsized":"2024-01-15T11:25:17.260517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:25:29.463134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118949+0000","last_change":"2024-01-15T11:25:15.238136+0000","last_active":"2024-01-15T11:25:17.118949+0000","last_peered":"2024-01-15T11:25:17.118949+0000","last_clean":"2024-01-15T11:25:17.118949+0000","last_became_active":"2024-01-15T11:25:15.237522+0000","last_became_peered":"2024-01-15T11:25:15.237522+0000","last_unstale":"2024-01-15T11:25:17.118949+0000","last_undegraded":"2024-01-15T11:25:17.118949+0000","last_fullsized":"2024-01-15T11:25:17.118949+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:16.499396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.261247+0000","last_change":"2024-01-15T11:25:16.263503+0000","last_active":"2024-01-15T11:25:17.261247+0000","last_peered":"2024-01-15T11:25:17.261247+0000","last_clean":"2024-01-15T11:25:17.261247+0000","last_became_active":"2024-01-15T11:25:16.263208+0000","last_became_peered":"2024-01-15T11:25:16.263208+0000","last_unstale":"2024-01-15T11:25:17.261247+0000","last_undegraded":"2024-01-15T11:25:17.261247+0000","last_fullsized":"2024-01-15T11:25:17.261247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:32:22.797112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.124941+0000","last_change":"2024-01-15T11:25:15.229209+0000","last_active":"2024-01-15T11:25:17.124941+0000","last_peered":"2024-01-15T11:25:17.124941+0000","last_clean":"2024-01-15T11:25:17.124941+0000","last_became_active":"2024-01-15T11:25:15.228899+0000","last_became_peered":"2024-01-15T11:25:15.228899+0000","last_unstale":"2024-01-15T11:25:17.124941+0000","last_undegraded":"2024-01-15T11:25:17.124941+0000","last_fullsized":"2024-01-15T11:25:17.124941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:56:32.006130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870812+0000","last_change":"2024-01-15T11:25:16.259973+0000","last_active":"2024-01-15T11:25:18.870812+0000","last_peered":"2024-01-15T11:25:18.870812+0000","last_clean":"2024-01-15T11:25:18.870812+0000","last_became_active":"2024-01-15T11:25:16.259589+0000","last_became_peered":"2024-01-15T11:25:16.259589+0000","last_unstale":"2024-01-15T11:25:18.870812+0000","last_undegraded":"2024-01-15T11:25:18.870812+0000","last_fullsized":"2024-01-15T11:25:18.870812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:28:28.895688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.128794+0000","last_change":"2024-01-15T11:25:15.234071+0000","last_active":"2024-01-15T11:25:17.128794+0000","last_peered":"2024-01-15T11:25:17.128794+0000","last_clean":"2024-01-15T11:25:17.128794+0000","last_became_active":"2024-01-15T11:25:15.233460+0000","last_became_peered":"2024-01-15T11:25:15.233460+0000","last_unstale":"2024-01-15T11:25:17.128794+0000","last_undegraded":"2024-01-15T11:25:17.128794+0000","last_fullsized":"2024-01-15T11:25:17.128794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:30.739157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.261956+0000","last_change":"2024-01-15T11:25:16.268001+0000","last_active":"2024-01-15T11:25:17.261956+0000","last_peered":"2024-01-15T11:25:17.261956+0000","last_clean":"2024-01-15T11:25:17.261956+0000","last_became_active":"2024-01-15T11:25:16.267277+0000","last_became_peered":"2024-01-15T11:25:16.267277+0000","last_unstale":"2024-01-15T11:25:17.261956+0000","last_undegraded":"2024-01-15T11:25:17.261956+0000","last_fullsized":"2024-01-15T11:25:17.261956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.255262+0000","last_change":"2024-01-15T11:25:16.258019+0000","last_active":"2024-01-15T11:25:17.255262+0000","last_peered":"2024-01-15T11:25:17.255262+0000","last_clean":"2024-01-15T11:25:17.255262+0000","last_became_active":"2024-01-15T11:25:16.257755+0000","last_became_peered":"2024-01-15T11:25:16.257755+0000","last_unstale":"2024-01-15T11:25:17.255262+0000","last_undegraded":"2024-01-15T11:25:17.255262+0000","last_fullsized":"2024-01-15T11:25:17.255262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:17:27.965755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121494+0000","last_change":"2024-01-15T11:25:15.235627+0000","last_active":"2024-01-15T11:25:17.121494+0000","last_peered":"2024-01-15T11:25:17.121494+0000","last_clean":"2024-01-15T11:25:17.121494+0000","last_became_active":"2024-01-15T11:25:15.235308+0000","last_became_peered":"2024-01-15T11:25:15.235308+0000","last_unstale":"2024-01-15T11:25:17.121494+0000","last_undegraded":"2024-01-15T11:25:17.121494+0000","last_fullsized":"2024-01-15T11:25:17.121494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:58.480006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870354+0000","last_change":"2024-01-15T11:25:16.260147+0000","last_active":"2024-01-15T11:25:18.870354+0000","last_peered":"2024-01-15T11:25:18.870354+0000","last_clean":"2024-01-15T11:25:18.870354+0000","last_became_active":"2024-01-15T11:25:16.259836+0000","last_became_peered":"2024-01-15T11:25:16.259836+0000","last_unstale":"2024-01-15T11:25:18.870354+0000","last_undegraded":"2024-01-15T11:25:18.870354+0000","last_fullsized":"2024-01-15T11:25:18.870354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:28:18.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.128819+0000","last_change":"2024-01-15T11:25:15.233565+0000","last_active":"2024-01-15T11:25:17.128819+0000","last_peered":"2024-01-15T11:25:17.128819+0000","last_clean":"2024-01-15T11:25:17.128819+0000","last_became_active":"2024-01-15T11:25:15.231919+0000","last_became_peered":"2024-01-15T11:25:15.231919+0000","last_unstale":"2024-01-15T11:25:17.128819+0000","last_undegraded":"2024-01-15T11:25:17.128819+0000","last_fullsized":"2024-01-15T11:25:17.128819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.243261+0000","last_change":"2024-01-15T11:25:03.506154+0000","last_active":"2024-01-15T11:25:16.243261+0000","last_peered":"2024-01-15T11:25:16.243261+0000","last_clean":"2024-01-15T11:25:16.243261+0000","last_became_active":"2024-01-15T11:25:03.504996+0000","last_became_peered":"2024-01-15T11:25:03.504996+0000","last_unstale":"2024-01-15T11:25:16.243261+0000","last_undegraded":"2024-01-15T11:25:16.243261+0000","last_fullsized":"2024-01-15T11:25:16.243261+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:02.494136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:02.494136+0000","last_clean_scrub_stamp":"2024-01-15T11:25:02.494136+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:22:33.043449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836254+0000","last_change":"2024-01-15T11:25:16.270301+0000","last_active":"2024-01-15T11:25:18.836254+0000","last_peered":"2024-01-15T11:25:18.836254+0000","last_clean":"2024-01-15T11:25:18.836254+0000","last_became_active":"2024-01-15T11:25:16.269969+0000","last_became_peered":"2024-01-15T11:25:16.269969+0000","last_unstale":"2024-01-15T11:25:18.836254+0000","last_undegraded":"2024-01-15T11:25:18.836254+0000","last_fullsized":"2024-01-15T11:25:18.836254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:54.759984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118117+0000","last_change":"2024-01-15T11:25:15.233382+0000","last_active":"2024-01-15T11:25:17.118117+0000","last_peered":"2024-01-15T11:25:17.118117+0000","last_clean":"2024-01-15T11:25:17.118117+0000","last_became_active":"2024-01-15T11:25:15.233111+0000","last_became_peered":"2024-01-15T11:25:15.233111+0000","last_unstale":"2024-01-15T11:25:17.118117+0000","last_undegraded":"2024-01-15T11:25:17.118117+0000","last_fullsized":"2024-01-15T11:25:17.118117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:50.865963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118244+0000","last_change":"2024-01-15T11:25:15.237505+0000","last_active":"2024-01-15T11:25:17.118244+0000","last_peered":"2024-01-15T11:25:17.118244+0000","last_clean":"2024-01-15T11:25:17.118244+0000","last_became_active":"2024-01-15T11:25:15.237206+0000","last_became_peered":"2024-01-15T11:25:15.237206+0000","last_unstale":"2024-01-15T11:25:17.118244+0000","last_undegraded":"2024-01-15T11:25:17.118244+0000","last_fullsized":"2024-01-15T11:25:17.118244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:14.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.255424+0000","last_change":"2024-01-15T11:25:16.254087+0000","last_active":"2024-01-15T11:25:17.255424+0000","last_peered":"2024-01-15T11:25:17.255424+0000","last_clean":"2024-01-15T11:25:17.255424+0000","last_became_active":"2024-01-15T11:25:16.253790+0000","last_became_peered":"2024-01-15T11:25:16.253790+0000","last_unstale":"2024-01-15T11:25:17.255424+0000","last_undegraded":"2024-01-15T11:25:17.255424+0000","last_fullsized":"2024-01-15T11:25:17.255424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:31.743954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.255927+0000","last_change":"2024-01-15T11:25:15.240170+0000","last_active":"2024-01-15T11:25:17.255927+0000","last_peered":"2024-01-15T11:25:17.255927+0000","last_clean":"2024-01-15T11:25:17.255927+0000","last_became_active":"2024-01-15T11:25:15.239842+0000","last_became_peered":"2024-01-15T11:25:15.239842+0000","last_unstale":"2024-01-15T11:25:17.255927+0000","last_undegraded":"2024-01-15T11:25:17.255927+0000","last_fullsized":"2024-01-15T11:25:17.255927+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:25.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870529+0000","last_change":"2024-01-15T11:25:16.247896+0000","last_active":"2024-01-15T11:25:18.870529+0000","last_peered":"2024-01-15T11:25:18.870529+0000","last_clean":"2024-01-15T11:25:18.870529+0000","last_became_active":"2024-01-15T11:25:16.247606+0000","last_became_peered":"2024-01-15T11:25:16.247606+0000","last_unstale":"2024-01-15T11:25:18.870529+0000","last_undegraded":"2024-01-15T11:25:18.870529+0000","last_fullsized":"2024-01-15T11:25:18.870529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:39:35.147679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.125401+0000","last_change":"2024-01-15T11:25:15.233571+0000","last_active":"2024-01-15T11:25:17.125401+0000","last_peered":"2024-01-15T11:25:17.125401+0000","last_clean":"2024-01-15T11:25:17.125401+0000","last_became_active":"2024-01-15T11:25:15.233180+0000","last_became_peered":"2024-01-15T11:25:15.233180+0000","last_unstale":"2024-01-15T11:25:17.125401+0000","last_undegraded":"2024-01-15T11:25:17.125401+0000","last_fullsized":"2024-01-15T11:25:17.125401+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:53.755186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836582+0000","last_change":"2024-01-15T11:25:16.253976+0000","last_active":"2024-01-15T11:25:18.836582+0000","last_peered":"2024-01-15T11:25:18.836582+0000","last_clean":"2024-01-15T11:25:18.836582+0000","last_became_active":"2024-01-15T11:25:16.253600+0000","last_became_peered":"2024-01-15T11:25:16.253600+0000","last_unstale":"2024-01-15T11:25:18.836582+0000","last_undegraded":"2024-01-15T11:25:18.836582+0000","last_fullsized":"2024-01-15T11:25:18.836582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:28:18.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118071+0000","last_change":"2024-01-15T11:25:15.235253+0000","last_active":"2024-01-15T11:25:17.118071+0000","last_peered":"2024-01-15T11:25:17.118071+0000","last_clean":"2024-01-15T11:25:17.118071+0000","last_became_active":"2024-01-15T11:25:15.234844+0000","last_became_peered":"2024-01-15T11:25:15.234844+0000","last_unstale":"2024-01-15T11:25:17.118071+0000","last_undegraded":"2024-01-15T11:25:17.118071+0000","last_fullsized":"2024-01-15T11:25:17.118071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:18.628480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.261714+0000","last_change":"2024-01-15T11:25:16.257934+0000","last_active":"2024-01-15T11:25:17.261714+0000","last_peered":"2024-01-15T11:25:17.261714+0000","last_clean":"2024-01-15T11:25:17.261714+0000","last_became_active":"2024-01-15T11:25:16.257566+0000","last_became_peered":"2024-01-15T11:25:16.257566+0000","last_unstale":"2024-01-15T11:25:17.261714+0000","last_undegraded":"2024-01-15T11:25:17.261714+0000","last_fullsized":"2024-01-15T11:25:17.261714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.124871+0000","last_change":"2024-01-15T11:25:15.237581+0000","last_active":"2024-01-15T11:25:17.124871+0000","last_peered":"2024-01-15T11:25:17.124871+0000","last_clean":"2024-01-15T11:25:17.124871+0000","last_became_active":"2024-01-15T11:25:15.237319+0000","last_became_peered":"2024-01-15T11:25:15.237319+0000","last_unstale":"2024-01-15T11:25:17.124871+0000","last_undegraded":"2024-01-15T11:25:17.124871+0000","last_fullsized":"2024-01-15T11:25:17.124871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870042+0000","last_change":"2024-01-15T11:25:16.256476+0000","last_active":"2024-01-15T11:25:18.870042+0000","last_peered":"2024-01-15T11:25:18.870042+0000","last_clean":"2024-01-15T11:25:18.870042+0000","last_became_active":"2024-01-15T11:25:16.256198+0000","last_became_peered":"2024-01-15T11:25:16.256198+0000","last_unstale":"2024-01-15T11:25:18.870042+0000","last_undegraded":"2024-01-15T11:25:18.870042+0000","last_fullsized":"2024-01-15T11:25:18.870042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:17:27.965755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.124914+0000","last_change":"2024-01-15T11:25:15.235154+0000","last_active":"2024-01-15T11:25:17.124914+0000","last_peered":"2024-01-15T11:25:17.124914+0000","last_clean":"2024-01-15T11:25:17.124914+0000","last_became_active":"2024-01-15T11:25:15.234870+0000","last_became_peered":"2024-01-15T11:25:15.234870+0000","last_unstale":"2024-01-15T11:25:17.124914+0000","last_undegraded":"2024-01-15T11:25:17.124914+0000","last_fullsized":"2024-01-15T11:25:17.124914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:53.755186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870250+0000","last_change":"2024-01-15T11:25:16.254780+0000","last_active":"2024-01-15T11:25:18.870250+0000","last_peered":"2024-01-15T11:25:18.870250+0000","last_clean":"2024-01-15T11:25:18.870250+0000","last_became_active":"2024-01-15T11:25:16.254519+0000","last_became_peered":"2024-01-15T11:25:16.254519+0000","last_unstale":"2024-01-15T11:25:18.870250+0000","last_undegraded":"2024-01-15T11:25:18.870250+0000","last_fullsized":"2024-01-15T11:25:18.870250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:15.719082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118181+0000","last_change":"2024-01-15T11:25:15.238538+0000","last_active":"2024-01-15T11:25:17.118181+0000","last_peered":"2024-01-15T11:25:17.118181+0000","last_clean":"2024-01-15T11:25:17.118181+0000","last_became_active":"2024-01-15T11:25:15.238222+0000","last_became_peered":"2024-01-15T11:25:15.238222+0000","last_unstale":"2024-01-15T11:25:17.118181+0000","last_undegraded":"2024-01-15T11:25:17.118181+0000","last_fullsized":"2024-01-15T11:25:17.118181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:30.739157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.281109+0000","last_change":"2024-01-15T11:25:16.262608+0000","last_active":"2024-01-15T11:25:18.281109+0000","last_peered":"2024-01-15T11:25:18.281109+0000","last_clean":"2024-01-15T11:25:18.281109+0000","last_became_active":"2024-01-15T11:25:16.262318+0000","last_became_peered":"2024-01-15T11:25:16.262318+0000","last_unstale":"2024-01-15T11:25:18.281109+0000","last_undegraded":"2024-01-15T11:25:18.281109+0000","last_fullsized":"2024-01-15T11:25:18.281109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:15.719082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121516+0000","last_change":"2024-01-15T11:25:15.232704+0000","last_active":"2024-01-15T11:25:17.121516+0000","last_peered":"2024-01-15T11:25:17.121516+0000","last_clean":"2024-01-15T11:25:17.121516+0000","last_became_active":"2024-01-15T11:25:15.232191+0000","last_became_peered":"2024-01-15T11:25:15.232191+0000","last_unstale":"2024-01-15T11:25:17.121516+0000","last_undegraded":"2024-01-15T11:25:17.121516+0000","last_fullsized":"2024-01-15T11:25:17.121516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:16.499396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260973+0000","last_change":"2024-01-15T11:25:16.260276+0000","last_active":"2024-01-15T11:25:17.260973+0000","last_peered":"2024-01-15T11:25:17.260973+0000","last_clean":"2024-01-15T11:25:17.260973+0000","last_became_active":"2024-01-15T11:25:16.259962+0000","last_became_peered":"2024-01-15T11:25:16.259962+0000","last_unstale":"2024-01-15T11:25:17.260973+0000","last_undegraded":"2024-01-15T11:25:17.260973+0000","last_fullsized":"2024-01-15T11:25:17.260973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:18:26.824601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.124968+0000","last_change":"2024-01-15T11:25:15.230446+0000","last_active":"2024-01-15T11:25:17.124968+0000","last_peered":"2024-01-15T11:25:17.124968+0000","last_clean":"2024-01-15T11:25:17.124968+0000","last_became_active":"2024-01-15T11:25:15.230176+0000","last_became_peered":"2024-01-15T11:25:15.230176+0000","last_unstale":"2024-01-15T11:25:17.124968+0000","last_undegraded":"2024-01-15T11:25:17.124968+0000","last_fullsized":"2024-01-15T11:25:17.124968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:32.421454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870752+0000","last_change":"2024-01-15T11:25:16.262438+0000","last_active":"2024-01-15T11:25:18.870752+0000","last_peered":"2024-01-15T11:25:18.870752+0000","last_clean":"2024-01-15T11:25:18.870752+0000","last_became_active":"2024-01-15T11:25:16.261849+0000","last_became_peered":"2024-01-15T11:25:16.261849+0000","last_unstale":"2024-01-15T11:25:18.870752+0000","last_undegraded":"2024-01-15T11:25:18.870752+0000","last_fullsized":"2024-01-15T11:25:18.870752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:25:29.463134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121555+0000","last_change":"2024-01-15T11:25:15.235007+0000","last_active":"2024-01-15T11:25:17.121555+0000","last_peered":"2024-01-15T11:25:17.121555+0000","last_clean":"2024-01-15T11:25:17.121555+0000","last_became_active":"2024-01-15T11:25:15.234709+0000","last_became_peered":"2024-01-15T11:25:15.234709+0000","last_unstale":"2024-01-15T11:25:17.121555+0000","last_undegraded":"2024-01-15T11:25:17.121555+0000","last_fullsized":"2024-01-15T11:25:17.121555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:25.427111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.262727+0000","last_change":"2024-01-15T11:25:16.263407+0000","last_active":"2024-01-15T11:25:17.262727+0000","last_peered":"2024-01-15T11:25:17.262727+0000","last_clean":"2024-01-15T11:25:17.262727+0000","last_became_active":"2024-01-15T11:25:16.263120+0000","last_became_peered":"2024-01-15T11:25:16.263120+0000","last_unstale":"2024-01-15T11:25:17.262727+0000","last_undegraded":"2024-01-15T11:25:17.262727+0000","last_fullsized":"2024-01-15T11:25:17.262727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:46:07.070505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.280889+0000","last_change":"2024-01-15T11:25:16.258910+0000","last_active":"2024-01-15T11:25:18.280889+0000","last_peered":"2024-01-15T11:25:18.280889+0000","last_clean":"2024-01-15T11:25:18.280889+0000","last_became_active":"2024-01-15T11:25:16.258619+0000","last_became_peered":"2024-01-15T11:25:16.258619+0000","last_unstale":"2024-01-15T11:25:18.280889+0000","last_undegraded":"2024-01-15T11:25:18.280889+0000","last_fullsized":"2024-01-15T11:25:18.280889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:02:52.680406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870231+0000","last_change":"2024-01-15T11:25:16.259468+0000","last_active":"2024-01-15T11:25:18.870231+0000","last_peered":"2024-01-15T11:25:18.870231+0000","last_clean":"2024-01-15T11:25:18.870231+0000","last_became_active":"2024-01-15T11:25:16.258703+0000","last_became_peered":"2024-01-15T11:25:16.258703+0000","last_unstale":"2024-01-15T11:25:18.870231+0000","last_undegraded":"2024-01-15T11:25:18.870231+0000","last_fullsized":"2024-01-15T11:25:18.870231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870367+0000","last_change":"2024-01-15T11:25:16.266445+0000","last_active":"2024-01-15T11:25:18.870367+0000","last_peered":"2024-01-15T11:25:18.870367+0000","last_clean":"2024-01-15T11:25:18.870367+0000","last_became_active":"2024-01-15T11:25:16.265486+0000","last_became_peered":"2024-01-15T11:25:16.265486+0000","last_unstale":"2024-01-15T11:25:18.870367+0000","last_undegraded":"2024-01-15T11:25:18.870367+0000","last_fullsized":"2024-01-15T11:25:18.870367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:08:06.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836503+0000","last_change":"2024-01-15T11:25:16.259210+0000","last_active":"2024-01-15T11:25:18.836503+0000","last_peered":"2024-01-15T11:25:18.836503+0000","last_clean":"2024-01-15T11:25:18.836503+0000","last_became_active":"2024-01-15T11:25:16.258919+0000","last_became_peered":"2024-01-15T11:25:16.258919+0000","last_unstale":"2024-01-15T11:25:18.836503+0000","last_undegraded":"2024-01-15T11:25:18.836503+0000","last_fullsized":"2024-01-15T11:25:18.836503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:15.719082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870445+0000","last_change":"2024-01-15T11:25:16.264604+0000","last_active":"2024-01-15T11:25:18.870445+0000","last_peered":"2024-01-15T11:25:18.870445+0000","last_clean":"2024-01-15T11:25:18.870445+0000","last_became_active":"2024-01-15T11:25:16.263950+0000","last_became_peered":"2024-01-15T11:25:16.263950+0000","last_unstale":"2024-01-15T11:25:18.870445+0000","last_undegraded":"2024-01-15T11:25:18.870445+0000","last_fullsized":"2024-01-15T11:25:18.870445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:15:35.970176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.255642+0000","last_change":"2024-01-15T11:25:16.256747+0000","last_active":"2024-01-15T11:25:17.255642+0000","last_peered":"2024-01-15T11:25:17.255642+0000","last_clean":"2024-01-15T11:25:17.255642+0000","last_became_active":"2024-01-15T11:25:16.256464+0000","last_became_peered":"2024-01-15T11:25:16.256464+0000","last_unstale":"2024-01-15T11:25:17.255642+0000","last_undegraded":"2024-01-15T11:25:17.255642+0000","last_fullsized":"2024-01-15T11:25:17.255642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:28:18.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.281343+0000","last_change":"2024-01-15T11:25:16.268099+0000","last_active":"2024-01-15T11:25:18.281343+0000","last_peered":"2024-01-15T11:25:18.281343+0000","last_clean":"2024-01-15T11:25:18.281343+0000","last_became_active":"2024-01-15T11:25:16.267826+0000","last_became_peered":"2024-01-15T11:25:16.267826+0000","last_unstale":"2024-01-15T11:25:18.281343+0000","last_undegraded":"2024-01-15T11:25:18.281343+0000","last_fullsized":"2024-01-15T11:25:18.281343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:15:35.970176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870390+0000","last_change":"2024-01-15T11:25:16.263641+0000","last_active":"2024-01-15T11:25:18.870390+0000","last_peered":"2024-01-15T11:25:18.870390+0000","last_clean":"2024-01-15T11:25:18.870390+0000","last_became_active":"2024-01-15T11:25:16.263368+0000","last_became_peered":"2024-01-15T11:25:16.263368+0000","last_unstale":"2024-01-15T11:25:18.870390+0000","last_undegraded":"2024-01-15T11:25:18.870390+0000","last_fullsized":"2024-01-15T11:25:18.870390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:49.784954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870273+0000","last_change":"2024-01-15T11:25:16.262170+0000","last_active":"2024-01-15T11:25:18.870273+0000","last_peered":"2024-01-15T11:25:18.870273+0000","last_clean":"2024-01-15T11:25:18.870273+0000","last_became_active":"2024-01-15T11:25:16.261695+0000","last_became_peered":"2024-01-15T11:25:16.261695+0000","last_unstale":"2024-01-15T11:25:18.870273+0000","last_undegraded":"2024-01-15T11:25:18.870273+0000","last_fullsized":"2024-01-15T11:25:18.870273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:49.784954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870152+0000","last_change":"2024-01-15T11:25:16.265406+0000","last_active":"2024-01-15T11:25:18.870152+0000","last_peered":"2024-01-15T11:25:18.870152+0000","last_clean":"2024-01-15T11:25:18.870152+0000","last_became_active":"2024-01-15T11:25:16.265150+0000","last_became_peered":"2024-01-15T11:25:16.265150+0000","last_unstale":"2024-01-15T11:25:18.870152+0000","last_undegraded":"2024-01-15T11:25:18.870152+0000","last_fullsized":"2024-01-15T11:25:18.870152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:02:52.680406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870317+0000","last_change":"2024-01-15T11:25:16.271625+0000","last_active":"2024-01-15T11:25:18.870317+0000","last_peered":"2024-01-15T11:25:18.870317+0000","last_clean":"2024-01-15T11:25:18.870317+0000","last_became_active":"2024-01-15T11:25:16.271361+0000","last_became_peered":"2024-01-15T11:25:16.271361+0000","last_unstale":"2024-01-15T11:25:18.870317+0000","last_undegraded":"2024-01-15T11:25:18.870317+0000","last_fullsized":"2024-01-15T11:25:18.870317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:15:35.970176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836444+0000","last_change":"2024-01-15T11:25:16.271113+0000","last_active":"2024-01-15T11:25:18.836444+0000","last_peered":"2024-01-15T11:25:18.836444+0000","last_clean":"2024-01-15T11:25:18.836444+0000","last_became_active":"2024-01-15T11:25:16.270795+0000","last_became_peered":"2024-01-15T11:25:16.270795+0000","last_unstale":"2024-01-15T11:25:18.836444+0000","last_undegraded":"2024-01-15T11:25:18.836444+0000","last_fullsized":"2024-01-15T11:25:18.836444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:02:52.680406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.281234+0000","last_change":"2024-01-15T11:25:16.262518+0000","last_active":"2024-01-15T11:25:18.281234+0000","last_peered":"2024-01-15T11:25:18.281234+0000","last_clean":"2024-01-15T11:25:18.281234+0000","last_became_active":"2024-01-15T11:25:16.262049+0000","last_became_peered":"2024-01-15T11:25:16.262049+0000","last_unstale":"2024-01-15T11:25:18.281234+0000","last_undegraded":"2024-01-15T11:25:18.281234+0000","last_fullsized":"2024-01-15T11:25:18.281234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:49.784954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T11:25:25.992 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T11:25:26.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.151+0000 7f77c538f700 1 Processor -- start 2024-01-15T11:25:26.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.151+0000 7f77c538f700 1 -- start start 2024-01-15T11:25:26.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.152+0000 7f77c538f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f77c015c160 0x7f77c00622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.152+0000 7f77c538f700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c015c540 con 0x7f77c00b04b0 2024-01-15T11:25:26.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.152+0000 7f77c538f700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c00b2ee0 con 0x7f77c00b2870 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.152+0000 7f77c538f700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c00add40 con 0x7f77c015c160 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.152+0000 7f77beffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f77c015c160 0x7f77c00622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.152+0000 7f77bf7fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f77c00b04b0 0x7f77c0165c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56614/0 (socket says 172.21.15.113:56614) 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.152+0000 7f77bf7fe700 1 -- 172.21.15.113:0/2561312923 learned_addr learned my addr 172.21.15.113:0/2561312923 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.152+0000 7f77bdffb700 1 -- 172.21.15.113:0/2561312923 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932992073 0 0) 0x7f77c015c540 con 0x7f77c00b04b0 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.152+0000 7f77bdffb700 1 -- 172.21.15.113:0/2561312923 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a80035b0 con 0x7f77c00b04b0 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.152+0000 7f77beffd700 1 -- 172.21.15.113:0/2561312923 >> v1:172.21.15.141:6789/0 conn(0x7f77c00b2870 legacy=0x7f77c00afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.152+0000 7f77beffd700 1 -- 172.21.15.113:0/2561312923 >> v1:172.21.15.113:6789/0 conn(0x7f77c00b04b0 legacy=0x7f77c0165c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.152+0000 7f77beffd700 1 -- 172.21.15.113:0/2561312923 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c00ad410 con 0x7f77c015c160 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.153+0000 7f77beffd700 1 --2- 172.21.15.113:0/2561312923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77c015c160 0x7f77c00622a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f77b4009840 tx=0x7f77b4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d98014336980326 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.153+0000 7f77bdffb700 1 -- 172.21.15.113:0/2561312923 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b4004230 con 0x7f77c015c160 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.153+0000 7f77bdffb700 1 -- 172.21.15.113:0/2561312923 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b4004390 con 0x7f77c015c160 2024-01-15T11:25:26.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.153+0000 7f77bdffb700 1 -- 172.21.15.113:0/2561312923 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b40036c0 con 0x7f77c015c160 2024-01-15T11:25:26.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.153+0000 7f77c538f700 1 -- 172.21.15.113:0/2561312923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77c015c160 msgr2=0x7f77c00622a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.153+0000 7f77c538f700 1 --2- 172.21.15.113:0/2561312923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77c015c160 0x7f77c00622a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f77b4009840 tx=0x7f77b4009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:26.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.153+0000 7f77c538f700 1 -- 172.21.15.113:0/2561312923 shutdown_connections 2024-01-15T11:25:26.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.153+0000 7f77c538f700 1 --2- 172.21.15.113:0/2561312923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77c015c160 0x7f77c00622a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.153+0000 7f77c538f700 1 -- 172.21.15.113:0/2561312923 >> 172.21.15.113:0/2561312923 conn(0x7f77c00b57b0 msgr2=0x7f77c00b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:26.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.153+0000 7f77c538f700 1 -- 172.21.15.113:0/2561312923 shutdown_connections 2024-01-15T11:25:26.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.153+0000 7f77c538f700 1 -- 172.21.15.113:0/2561312923 wait complete. 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.155+0000 7f77c538f700 1 Processor -- start 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.155+0000 7f77c538f700 1 -- start start 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.155+0000 7f77c538f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77c00b04b0 0x7f77c0143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.155+0000 7f77c538f700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77c00b2870 0x7f77c0140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.155+0000 7f77c538f700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77c015c160 0x7f77c0151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.155+0000 7f77c538f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c0144350 con 0x7f77c00b04b0 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.155+0000 7f77c538f700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c01f5160 con 0x7f77c015c160 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.155+0000 7f77c538f700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c01f5440 con 0x7f77c00b2870 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77beffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77c00b04b0 0x7f77c0143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77beffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77c00b04b0 0x7f77c0143e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37874/0 (socket says 172.21.15.113:37874) 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77beffd700 1 -- 172.21.15.113:0/293088741 learned_addr learned my addr 172.21.15.113:0/293088741 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77bf7fe700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77c015c160 0x7f77c0151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77be7fc700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77c00b2870 0x7f77c0140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77beffd700 1 -- 172.21.15.113:0/293088741 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77c00b2870 msgr2=0x7f77c0140e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77beffd700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77c00b2870 0x7f77c0140e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77beffd700 1 -- 172.21.15.113:0/293088741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77c015c160 msgr2=0x7f77c0151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77beffd700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77c015c160 0x7f77c0151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77beffd700 1 -- 172.21.15.113:0/293088741 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b40094a0 con 0x7f77c00b04b0 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77be7fc700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77c00b2870 0x7f77c0140e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77bf7fe700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77c015c160 0x7f77c0151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77beffd700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77c00b04b0 0x7f77c0143e10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f77b40090b0 tx=0x7f77b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77a77fe700 1 -- 172.21.15.113:0/293088741 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b4011630 con 0x7f77c00b04b0 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77a77fe700 1 -- 172.21.15.113:0/293088741 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b4011790 con 0x7f77c00b04b0 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77c538f700 1 -- 172.21.15.113:0/293088741 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c01f5720 con 0x7f77c00b04b0 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77a77fe700 1 -- 172.21.15.113:0/293088741 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b4011b00 con 0x7f77c00b04b0 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.156+0000 7f77c538f700 1 -- 172.21.15.113:0/293088741 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c01413d0 con 0x7f77c00b04b0 2024-01-15T11:25:26.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.157+0000 7f77c538f700 1 -- 172.21.15.113:0/293088741 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c0067f60 con 0x7f77c00b04b0 2024-01-15T11:25:26.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.157+0000 7f77a77fe700 1 -- 172.21.15.113:0/293088741 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f77b4018040 con 0x7f77c00b04b0 2024-01-15T11:25:26.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.158+0000 7f77a77fe700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f77ac06b350 0x7f77ac06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.158+0000 7f77a77fe700 1 -- 172.21.15.113:0/293088741 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77b408a710 con 0x7f77c00b04b0 2024-01-15T11:25:26.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.158+0000 7f77be7fc700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f77ac06b350 0x7f77ac06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.158+0000 7f77be7fc700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f77ac06b350 0x7f77ac06d810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f77c015d5c0 tx=0x7f77b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:26.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.162+0000 7f77a77fe700 1 -- 172.21.15.113:0/293088741 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77b4055730 con 0x7f77c00b04b0 2024-01-15T11:25:26.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.349+0000 7f77c538f700 1 -- 172.21.15.113:0/293088741 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77c00ad9d0 con 0x7f77ac06b350 2024-01-15T11:25:26.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.357+0000 7f77a77fe700 1 -- 172.21.15.113:0/293088741 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264606 (secure 0 0 0) 0x7f77b0014420 con 0x7f77ac06b350 2024-01-15T11:25:26.359 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:26.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.360+0000 7f77c538f700 1 -- 172.21.15.113:0/293088741 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f77ac06b350 msgr2=0x7f77ac06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.360+0000 7f77c538f700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f77ac06b350 0x7f77ac06d810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f77c015d5c0 tx=0x7f77b0009040 comp rx=0 tx=0).stop 2024-01-15T11:25:26.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.360+0000 7f77c538f700 1 -- 172.21.15.113:0/293088741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77c00b04b0 msgr2=0x7f77c0143e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.360+0000 7f77c538f700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77c00b04b0 0x7f77c0143e10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f77b40090b0 tx=0x7f77b4004750 comp rx=0 tx=0).stop 2024-01-15T11:25:26.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.360+0000 7f77c538f700 1 -- 172.21.15.113:0/293088741 shutdown_connections 2024-01-15T11:25:26.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.360+0000 7f77c538f700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f77ac06b350 0x7f77ac06d810 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.360+0000 7f77c538f700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77c015c160 0x7f77c0151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.360+0000 7f77c538f700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77c00b04b0 0x7f77c0143e10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.360+0000 7f77c538f700 1 --2- 172.21.15.113:0/293088741 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f77c00b2870 0x7f77c0140e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.360+0000 7f77c538f700 1 -- 172.21.15.113:0/293088741 >> 172.21.15.113:0/293088741 conn(0x7f77c00b57b0 msgr2=0x7f77c00ceb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:26.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.360+0000 7f77c538f700 1 -- 172.21.15.113:0/293088741 shutdown_connections 2024-01-15T11:25:26.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.360+0000 7f77c538f700 1 -- 172.21.15.113:0/293088741 wait complete. 2024-01-15T11:25:26.363 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-15T11:25:26.371 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-15T11:25:26.111147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219008,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755712,"statfs":{"total":773094113280,"available":772869849088,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.869750"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.262398+0000","last_change":"2024-01-15T11:25:16.269790+0000","last_active":"2024-01-15T11:25:17.262398+0000","last_peered":"2024-01-15T11:25:17.262398+0000","last_clean":"2024-01-15T11:25:17.262398+0000","last_became_active":"2024-01-15T11:25:16.269540+0000","last_became_peered":"2024-01-15T11:25:16.269540+0000","last_unstale":"2024-01-15T11:25:17.262398+0000","last_undegraded":"2024-01-15T11:25:17.262398+0000","last_fullsized":"2024-01-15T11:25:17.262398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:47:05.456406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.256089+0000","last_change":"2024-01-15T11:25:16.254768+0000","last_active":"2024-01-15T11:25:17.256089+0000","last_peered":"2024-01-15T11:25:17.256089+0000","last_clean":"2024-01-15T11:25:17.256089+0000","last_became_active":"2024-01-15T11:25:16.254438+0000","last_became_peered":"2024-01-15T11:25:16.254438+0000","last_unstale":"2024-01-15T11:25:17.256089+0000","last_undegraded":"2024-01-15T11:25:17.256089+0000","last_fullsized":"2024-01-15T11:25:17.256089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:49.784954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836465+0000","last_change":"2024-01-15T11:25:16.255993+0000","last_active":"2024-01-15T11:25:18.836465+0000","last_peered":"2024-01-15T11:25:18.836465+0000","last_clean":"2024-01-15T11:25:18.836465+0000","last_became_active":"2024-01-15T11:25:16.255718+0000","last_became_peered":"2024-01-15T11:25:16.255718+0000","last_unstale":"2024-01-15T11:25:18.836465+0000","last_undegraded":"2024-01-15T11:25:18.836465+0000","last_fullsized":"2024-01-15T11:25:18.836465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:31.743954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870637+0000","last_change":"2024-01-15T11:25:16.262620+0000","last_active":"2024-01-15T11:25:18.870637+0000","last_peered":"2024-01-15T11:25:18.870637+0000","last_clean":"2024-01-15T11:25:18.870637+0000","last_became_active":"2024-01-15T11:25:16.262147+0000","last_became_peered":"2024-01-15T11:25:16.262147+0000","last_unstale":"2024-01-15T11:25:18.870637+0000","last_undegraded":"2024-01-15T11:25:18.870637+0000","last_fullsized":"2024-01-15T11:25:18.870637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.346395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260464+0000","last_change":"2024-01-15T11:25:16.252826+0000","last_active":"2024-01-15T11:25:17.260464+0000","last_peered":"2024-01-15T11:25:17.260464+0000","last_clean":"2024-01-15T11:25:17.260464+0000","last_became_active":"2024-01-15T11:25:16.251976+0000","last_became_peered":"2024-01-15T11:25:16.251976+0000","last_unstale":"2024-01-15T11:25:17.260464+0000","last_undegraded":"2024-01-15T11:25:17.260464+0000","last_fullsized":"2024-01-15T11:25:17.260464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:15:36.291852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870521+0000","last_change":"2024-01-15T11:25:16.266650+0000","last_active":"2024-01-15T11:25:18.870521+0000","last_peered":"2024-01-15T11:25:18.870521+0000","last_clean":"2024-01-15T11:25:18.870521+0000","last_became_active":"2024-01-15T11:25:16.266157+0000","last_became_peered":"2024-01-15T11:25:16.266157+0000","last_unstale":"2024-01-15T11:25:18.870521+0000","last_undegraded":"2024-01-15T11:25:18.870521+0000","last_fullsized":"2024-01-15T11:25:18.870521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:39:35.147679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260275+0000","last_change":"2024-01-15T11:25:16.255535+0000","last_active":"2024-01-15T11:25:17.260275+0000","last_peered":"2024-01-15T11:25:17.260275+0000","last_clean":"2024-01-15T11:25:17.260275+0000","last_became_active":"2024-01-15T11:25:16.255248+0000","last_became_peered":"2024-01-15T11:25:16.255248+0000","last_unstale":"2024-01-15T11:25:17.260275+0000","last_undegraded":"2024-01-15T11:25:17.260275+0000","last_fullsized":"2024-01-15T11:25:17.260275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.346395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.869947+0000","last_change":"2024-01-15T11:25:16.258135+0000","last_active":"2024-01-15T11:25:18.869947+0000","last_peered":"2024-01-15T11:25:18.869947+0000","last_clean":"2024-01-15T11:25:18.869947+0000","last_became_active":"2024-01-15T11:25:16.257859+0000","last_became_peered":"2024-01-15T11:25:16.257859+0000","last_unstale":"2024-01-15T11:25:18.869947+0000","last_undegraded":"2024-01-15T11:25:18.869947+0000","last_fullsized":"2024-01-15T11:25:18.869947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:56:33.010927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.262562+0000","last_change":"2024-01-15T11:25:16.261234+0000","last_active":"2024-01-15T11:25:17.262562+0000","last_peered":"2024-01-15T11:25:17.262562+0000","last_clean":"2024-01-15T11:25:17.262562+0000","last_became_active":"2024-01-15T11:25:16.260952+0000","last_became_peered":"2024-01-15T11:25:16.260952+0000","last_unstale":"2024-01-15T11:25:17.262562+0000","last_undegraded":"2024-01-15T11:25:17.262562+0000","last_fullsized":"2024-01-15T11:25:17.262562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:22:29.117384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.281188+0000","last_change":"2024-01-15T11:25:16.266369+0000","last_active":"2024-01-15T11:25:18.281188+0000","last_peered":"2024-01-15T11:25:18.281188+0000","last_clean":"2024-01-15T11:25:18.281188+0000","last_became_active":"2024-01-15T11:25:16.266091+0000","last_became_peered":"2024-01-15T11:25:16.266091+0000","last_unstale":"2024-01-15T11:25:18.281188+0000","last_undegraded":"2024-01-15T11:25:18.281188+0000","last_fullsized":"2024-01-15T11:25:18.281188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:28:18.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.281095+0000","last_change":"2024-01-15T11:25:16.259582+0000","last_active":"2024-01-15T11:25:18.281095+0000","last_peered":"2024-01-15T11:25:18.281095+0000","last_clean":"2024-01-15T11:25:18.281095+0000","last_became_active":"2024-01-15T11:25:16.259270+0000","last_became_peered":"2024-01-15T11:25:16.259270+0000","last_unstale":"2024-01-15T11:25:18.281095+0000","last_undegraded":"2024-01-15T11:25:18.281095+0000","last_fullsized":"2024-01-15T11:25:18.281095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:31.743954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.262196+0000","last_change":"2024-01-15T11:25:16.274052+0000","last_active":"2024-01-15T11:25:17.262196+0000","last_peered":"2024-01-15T11:25:17.262196+0000","last_clean":"2024-01-15T11:25:17.262196+0000","last_became_active":"2024-01-15T11:25:16.273798+0000","last_became_peered":"2024-01-15T11:25:16.273798+0000","last_unstale":"2024-01-15T11:25:17.262196+0000","last_undegraded":"2024-01-15T11:25:17.262196+0000","last_fullsized":"2024-01-15T11:25:17.262196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:05.228308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870217+0000","last_change":"2024-01-15T11:25:16.250458+0000","last_active":"2024-01-15T11:25:18.870217+0000","last_peered":"2024-01-15T11:25:18.870217+0000","last_clean":"2024-01-15T11:25:18.870217+0000","last_became_active":"2024-01-15T11:25:16.250262+0000","last_became_peered":"2024-01-15T11:25:16.250262+0000","last_unstale":"2024-01-15T11:25:18.870217+0000","last_undegraded":"2024-01-15T11:25:18.870217+0000","last_fullsized":"2024-01-15T11:25:18.870217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:31.743954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836352+0000","last_change":"2024-01-15T11:25:16.253950+0000","last_active":"2024-01-15T11:25:18.836352+0000","last_peered":"2024-01-15T11:25:18.836352+0000","last_clean":"2024-01-15T11:25:18.836352+0000","last_became_active":"2024-01-15T11:25:16.253667+0000","last_became_peered":"2024-01-15T11:25:16.253667+0000","last_unstale":"2024-01-15T11:25:18.836352+0000","last_undegraded":"2024-01-15T11:25:18.836352+0000","last_fullsized":"2024-01-15T11:25:18.836352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:56:33.010927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.255356+0000","last_change":"2024-01-15T11:25:16.268387+0000","last_active":"2024-01-15T11:25:17.255356+0000","last_peered":"2024-01-15T11:25:17.255356+0000","last_clean":"2024-01-15T11:25:17.255356+0000","last_became_active":"2024-01-15T11:25:16.268088+0000","last_became_peered":"2024-01-15T11:25:16.268088+0000","last_unstale":"2024-01-15T11:25:17.255356+0000","last_undegraded":"2024-01-15T11:25:17.255356+0000","last_fullsized":"2024-01-15T11:25:17.255356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:02:52.680406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870265+0000","last_change":"2024-01-15T11:25:16.261989+0000","last_active":"2024-01-15T11:25:18.870265+0000","last_peered":"2024-01-15T11:25:18.870265+0000","last_clean":"2024-01-15T11:25:18.870265+0000","last_became_active":"2024-01-15T11:25:16.261260+0000","last_became_peered":"2024-01-15T11:25:16.261260+0000","last_unstale":"2024-01-15T11:25:18.870265+0000","last_undegraded":"2024-01-15T11:25:18.870265+0000","last_fullsized":"2024-01-15T11:25:18.870265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:28:18.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870959+0000","last_change":"2024-01-15T11:25:16.270904+0000","last_active":"2024-01-15T11:25:18.870959+0000","last_peered":"2024-01-15T11:25:18.870959+0000","last_clean":"2024-01-15T11:25:18.870959+0000","last_became_active":"2024-01-15T11:25:16.270611+0000","last_became_peered":"2024-01-15T11:25:16.270611+0000","last_unstale":"2024-01-15T11:25:18.870959+0000","last_undegraded":"2024-01-15T11:25:18.870959+0000","last_fullsized":"2024-01-15T11:25:18.870959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:18:26.824601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870399+0000","last_change":"2024-01-15T11:25:16.257019+0000","last_active":"2024-01-15T11:25:18.870399+0000","last_peered":"2024-01-15T11:25:18.870399+0000","last_clean":"2024-01-15T11:25:18.870399+0000","last_became_active":"2024-01-15T11:25:16.256278+0000","last_became_peered":"2024-01-15T11:25:16.256278+0000","last_unstale":"2024-01-15T11:25:18.870399+0000","last_undegraded":"2024-01-15T11:25:18.870399+0000","last_fullsized":"2024-01-15T11:25:18.870399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:05.228308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260725+0000","last_change":"2024-01-15T11:25:16.248798+0000","last_active":"2024-01-15T11:25:17.260725+0000","last_peered":"2024-01-15T11:25:17.260725+0000","last_clean":"2024-01-15T11:25:17.260725+0000","last_became_active":"2024-01-15T11:25:16.248488+0000","last_became_peered":"2024-01-15T11:25:16.248488+0000","last_unstale":"2024-01-15T11:25:17.260725+0000","last_undegraded":"2024-01-15T11:25:17.260725+0000","last_fullsized":"2024-01-15T11:25:17.260725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:37:51.629738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.148831+0000","last_change":"2024-01-15T11:25:15.233384+0000","last_active":"2024-01-15T11:25:17.148831+0000","last_peered":"2024-01-15T11:25:17.148831+0000","last_clean":"2024-01-15T11:25:17.148831+0000","last_became_active":"2024-01-15T11:25:15.233134+0000","last_became_peered":"2024-01-15T11:25:15.233134+0000","last_unstale":"2024-01-15T11:25:17.148831+0000","last_undegraded":"2024-01-15T11:25:17.148831+0000","last_fullsized":"2024-01-15T11:25:17.148831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870182+0000","last_change":"2024-01-15T11:25:16.256579+0000","last_active":"2024-01-15T11:25:18.870182+0000","last_peered":"2024-01-15T11:25:18.870182+0000","last_clean":"2024-01-15T11:25:18.870182+0000","last_became_active":"2024-01-15T11:25:16.256320+0000","last_became_peered":"2024-01-15T11:25:16.256320+0000","last_unstale":"2024-01-15T11:25:18.870182+0000","last_undegraded":"2024-01-15T11:25:18.870182+0000","last_fullsized":"2024-01-15T11:25:18.870182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.346395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121397+0000","last_change":"2024-01-15T11:25:15.230417+0000","last_active":"2024-01-15T11:25:17.121397+0000","last_peered":"2024-01-15T11:25:17.121397+0000","last_clean":"2024-01-15T11:25:17.121397+0000","last_became_active":"2024-01-15T11:25:15.230135+0000","last_became_peered":"2024-01-15T11:25:15.230135+0000","last_unstale":"2024-01-15T11:25:17.121397+0000","last_undegraded":"2024-01-15T11:25:17.121397+0000","last_fullsized":"2024-01-15T11:25:17.121397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:25.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836423+0000","last_change":"2024-01-15T11:25:16.259985+0000","last_active":"2024-01-15T11:25:18.836423+0000","last_peered":"2024-01-15T11:25:18.836423+0000","last_clean":"2024-01-15T11:25:18.836423+0000","last_became_active":"2024-01-15T11:25:16.259697+0000","last_became_peered":"2024-01-15T11:25:16.259697+0000","last_unstale":"2024-01-15T11:25:18.836423+0000","last_undegraded":"2024-01-15T11:25:18.836423+0000","last_fullsized":"2024-01-15T11:25:18.836423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:17:27.965755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118258+0000","last_change":"2024-01-15T11:25:15.238695+0000","last_active":"2024-01-15T11:25:17.118258+0000","last_peered":"2024-01-15T11:25:17.118258+0000","last_clean":"2024-01-15T11:25:17.118258+0000","last_became_active":"2024-01-15T11:25:15.238377+0000","last_became_peered":"2024-01-15T11:25:15.238377+0000","last_unstale":"2024-01-15T11:25:17.118258+0000","last_undegraded":"2024-01-15T11:25:17.118258+0000","last_fullsized":"2024-01-15T11:25:17.118258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:48.780156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870868+0000","last_change":"2024-01-15T11:25:16.264379+0000","last_active":"2024-01-15T11:25:18.870868+0000","last_peered":"2024-01-15T11:25:18.870868+0000","last_clean":"2024-01-15T11:25:18.870868+0000","last_became_active":"2024-01-15T11:25:16.264121+0000","last_became_peered":"2024-01-15T11:25:16.264121+0000","last_unstale":"2024-01-15T11:25:18.870868+0000","last_undegraded":"2024-01-15T11:25:18.870868+0000","last_fullsized":"2024-01-15T11:25:18.870868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:37:51.629738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118194+0000","last_change":"2024-01-15T11:25:15.237423+0000","last_active":"2024-01-15T11:25:17.118194+0000","last_peered":"2024-01-15T11:25:17.118194+0000","last_clean":"2024-01-15T11:25:17.118194+0000","last_became_active":"2024-01-15T11:25:15.237121+0000","last_became_peered":"2024-01-15T11:25:15.237121+0000","last_unstale":"2024-01-15T11:25:17.118194+0000","last_undegraded":"2024-01-15T11:25:17.118194+0000","last_fullsized":"2024-01-15T11:25:17.118194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:04.223510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870892+0000","last_change":"2024-01-15T11:25:16.266842+0000","last_active":"2024-01-15T11:25:18.870892+0000","last_peered":"2024-01-15T11:25:18.870892+0000","last_clean":"2024-01-15T11:25:18.870892+0000","last_became_active":"2024-01-15T11:25:16.266606+0000","last_became_peered":"2024-01-15T11:25:16.266606+0000","last_unstale":"2024-01-15T11:25:18.870892+0000","last_undegraded":"2024-01-15T11:25:18.870892+0000","last_fullsized":"2024-01-15T11:25:18.870892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:37.335593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121353+0000","last_change":"2024-01-15T11:25:15.232263+0000","last_active":"2024-01-15T11:25:17.121353+0000","last_peered":"2024-01-15T11:25:17.121353+0000","last_clean":"2024-01-15T11:25:17.121353+0000","last_became_active":"2024-01-15T11:25:15.231976+0000","last_became_peered":"2024-01-15T11:25:15.231976+0000","last_unstale":"2024-01-15T11:25:17.121353+0000","last_undegraded":"2024-01-15T11:25:17.121353+0000","last_fullsized":"2024-01-15T11:25:17.121353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:25.797063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.261726+0000","last_change":"2024-01-15T11:25:16.263430+0000","last_active":"2024-01-15T11:25:17.261726+0000","last_peered":"2024-01-15T11:25:17.261726+0000","last_clean":"2024-01-15T11:25:17.261726+0000","last_became_active":"2024-01-15T11:25:16.261944+0000","last_became_peered":"2024-01-15T11:25:16.261944+0000","last_unstale":"2024-01-15T11:25:17.261726+0000","last_undegraded":"2024-01-15T11:25:17.261726+0000","last_fullsized":"2024-01-15T11:25:17.261726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.431909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.125258+0000","last_change":"2024-01-15T11:25:15.235451+0000","last_active":"2024-01-15T11:25:17.125258+0000","last_peered":"2024-01-15T11:25:17.125258+0000","last_clean":"2024-01-15T11:25:17.125258+0000","last_became_active":"2024-01-15T11:25:15.235166+0000","last_became_peered":"2024-01-15T11:25:15.235166+0000","last_unstale":"2024-01-15T11:25:17.125258+0000","last_undegraded":"2024-01-15T11:25:17.125258+0000","last_fullsized":"2024-01-15T11:25:17.125258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836642+0000","last_change":"2024-01-15T11:25:16.262932+0000","last_active":"2024-01-15T11:25:18.836642+0000","last_peered":"2024-01-15T11:25:18.836642+0000","last_clean":"2024-01-15T11:25:18.836642+0000","last_became_active":"2024-01-15T11:25:16.262667+0000","last_became_peered":"2024-01-15T11:25:16.262667+0000","last_unstale":"2024-01-15T11:25:18.836642+0000","last_undegraded":"2024-01-15T11:25:18.836642+0000","last_fullsized":"2024-01-15T11:25:18.836642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:49.784954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121325+0000","last_change":"2024-01-15T11:25:15.232286+0000","last_active":"2024-01-15T11:25:17.121325+0000","last_peered":"2024-01-15T11:25:17.121325+0000","last_clean":"2024-01-15T11:25:17.121325+0000","last_became_active":"2024-01-15T11:25:15.230764+0000","last_became_peered":"2024-01-15T11:25:15.230764+0000","last_unstale":"2024-01-15T11:25:17.121325+0000","last_undegraded":"2024-01-15T11:25:17.121325+0000","last_fullsized":"2024-01-15T11:25:17.121325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:53.755186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870515+0000","last_change":"2024-01-15T11:25:16.254427+0000","last_active":"2024-01-15T11:25:18.870515+0000","last_peered":"2024-01-15T11:25:18.870515+0000","last_clean":"2024-01-15T11:25:18.870515+0000","last_became_active":"2024-01-15T11:25:16.253271+0000","last_became_peered":"2024-01-15T11:25:16.253271+0000","last_unstale":"2024-01-15T11:25:18.870515+0000","last_undegraded":"2024-01-15T11:25:18.870515+0000","last_fullsized":"2024-01-15T11:25:18.870515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:08:06.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.128744+0000","last_change":"2024-01-15T11:25:15.231253+0000","last_active":"2024-01-15T11:25:17.128744+0000","last_peered":"2024-01-15T11:25:17.128744+0000","last_clean":"2024-01-15T11:25:17.128744+0000","last_became_active":"2024-01-15T11:25:15.230935+0000","last_became_peered":"2024-01-15T11:25:15.230935+0000","last_unstale":"2024-01-15T11:25:17.128744+0000","last_undegraded":"2024-01-15T11:25:17.128744+0000","last_fullsized":"2024-01-15T11:25:17.128744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:04.223510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.255483+0000","last_change":"2024-01-15T11:25:16.256798+0000","last_active":"2024-01-15T11:25:17.255483+0000","last_peered":"2024-01-15T11:25:17.255483+0000","last_clean":"2024-01-15T11:25:17.255483+0000","last_became_active":"2024-01-15T11:25:16.256540+0000","last_became_peered":"2024-01-15T11:25:16.256540+0000","last_unstale":"2024-01-15T11:25:17.255483+0000","last_undegraded":"2024-01-15T11:25:17.255483+0000","last_fullsized":"2024-01-15T11:25:17.255483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:15.719082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.125424+0000","last_change":"2024-01-15T11:25:15.236440+0000","last_active":"2024-01-15T11:25:17.125424+0000","last_peered":"2024-01-15T11:25:17.125424+0000","last_clean":"2024-01-15T11:25:17.125424+0000","last_became_active":"2024-01-15T11:25:15.236034+0000","last_became_peered":"2024-01-15T11:25:15.236034+0000","last_unstale":"2024-01-15T11:25:17.125424+0000","last_undegraded":"2024-01-15T11:25:17.125424+0000","last_fullsized":"2024-01-15T11:25:17.125424+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:25.797063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870721+0000","last_change":"2024-01-15T11:25:16.270509+0000","last_active":"2024-01-15T11:25:18.870721+0000","last_peered":"2024-01-15T11:25:18.870721+0000","last_clean":"2024-01-15T11:25:18.870721+0000","last_became_active":"2024-01-15T11:25:16.270319+0000","last_became_peered":"2024-01-15T11:25:16.270319+0000","last_unstale":"2024-01-15T11:25:18.870721+0000","last_undegraded":"2024-01-15T11:25:18.870721+0000","last_fullsized":"2024-01-15T11:25:18.870721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:15:36.291852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.128892+0000","last_change":"2024-01-15T11:25:15.235314+0000","last_active":"2024-01-15T11:25:17.128892+0000","last_peered":"2024-01-15T11:25:17.128892+0000","last_clean":"2024-01-15T11:25:17.128892+0000","last_became_active":"2024-01-15T11:25:15.232515+0000","last_became_peered":"2024-01-15T11:25:15.232515+0000","last_unstale":"2024-01-15T11:25:17.128892+0000","last_undegraded":"2024-01-15T11:25:17.128892+0000","last_fullsized":"2024-01-15T11:25:17.128892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:32.421454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836642+0000","last_change":"2024-01-15T11:25:16.260223+0000","last_active":"2024-01-15T11:25:18.836642+0000","last_peered":"2024-01-15T11:25:18.836642+0000","last_clean":"2024-01-15T11:25:18.836642+0000","last_became_active":"2024-01-15T11:25:16.259365+0000","last_became_peered":"2024-01-15T11:25:16.259365+0000","last_unstale":"2024-01-15T11:25:18.836642+0000","last_undegraded":"2024-01-15T11:25:18.836642+0000","last_fullsized":"2024-01-15T11:25:18.836642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:15:35.970176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.124986+0000","last_change":"2024-01-15T11:25:15.233433+0000","last_active":"2024-01-15T11:25:17.124986+0000","last_peered":"2024-01-15T11:25:17.124986+0000","last_clean":"2024-01-15T11:25:17.124986+0000","last_became_active":"2024-01-15T11:25:15.233046+0000","last_became_peered":"2024-01-15T11:25:15.233046+0000","last_unstale":"2024-01-15T11:25:17.124986+0000","last_undegraded":"2024-01-15T11:25:17.124986+0000","last_fullsized":"2024-01-15T11:25:17.124986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:25.797063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.262050+0000","last_change":"2024-01-15T11:25:16.275023+0000","last_active":"2024-01-15T11:25:17.262050+0000","last_peered":"2024-01-15T11:25:17.262050+0000","last_clean":"2024-01-15T11:25:17.262050+0000","last_became_active":"2024-01-15T11:25:16.274744+0000","last_became_peered":"2024-01-15T11:25:16.274744+0000","last_unstale":"2024-01-15T11:25:17.262050+0000","last_undegraded":"2024-01-15T11:25:17.262050+0000","last_fullsized":"2024-01-15T11:25:17.262050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.125282+0000","last_change":"2024-01-15T11:25:15.235460+0000","last_active":"2024-01-15T11:25:17.125282+0000","last_peered":"2024-01-15T11:25:17.125282+0000","last_clean":"2024-01-15T11:25:17.125282+0000","last_became_active":"2024-01-15T11:25:15.235194+0000","last_became_peered":"2024-01-15T11:25:15.235194+0000","last_unstale":"2024-01-15T11:25:17.125282+0000","last_undegraded":"2024-01-15T11:25:17.125282+0000","last_fullsized":"2024-01-15T11:25:17.125282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:32.421454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260798+0000","last_change":"2024-01-15T11:25:16.254349+0000","last_active":"2024-01-15T11:25:17.260798+0000","last_peered":"2024-01-15T11:25:17.260798+0000","last_clean":"2024-01-15T11:25:17.260798+0000","last_became_active":"2024-01-15T11:25:16.254068+0000","last_became_peered":"2024-01-15T11:25:16.254068+0000","last_unstale":"2024-01-15T11:25:17.260798+0000","last_undegraded":"2024-01-15T11:25:17.260798+0000","last_fullsized":"2024-01-15T11:25:17.260798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:37.335593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.148740+0000","last_change":"2024-01-15T11:25:15.236870+0000","last_active":"2024-01-15T11:25:17.148740+0000","last_peered":"2024-01-15T11:25:17.148740+0000","last_clean":"2024-01-15T11:25:17.148740+0000","last_became_active":"2024-01-15T11:25:15.236597+0000","last_became_peered":"2024-01-15T11:25:15.236597+0000","last_unstale":"2024-01-15T11:25:17.148740+0000","last_undegraded":"2024-01-15T11:25:17.148740+0000","last_fullsized":"2024-01-15T11:25:17.148740+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:04.223510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836341+0000","last_change":"2024-01-15T11:25:16.257709+0000","last_active":"2024-01-15T11:25:18.836341+0000","last_peered":"2024-01-15T11:25:18.836341+0000","last_clean":"2024-01-15T11:25:18.836341+0000","last_became_active":"2024-01-15T11:25:16.257415+0000","last_became_peered":"2024-01-15T11:25:16.257415+0000","last_unstale":"2024-01-15T11:25:18.836341+0000","last_undegraded":"2024-01-15T11:25:18.836341+0000","last_fullsized":"2024-01-15T11:25:18.836341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:26.801860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.128807+0000","last_change":"2024-01-15T11:25:15.233904+0000","last_active":"2024-01-15T11:25:17.128807+0000","last_peered":"2024-01-15T11:25:17.128807+0000","last_clean":"2024-01-15T11:25:17.128807+0000","last_became_active":"2024-01-15T11:25:15.232303+0000","last_became_peered":"2024-01-15T11:25:15.232303+0000","last_unstale":"2024-01-15T11:25:17.128807+0000","last_undegraded":"2024-01-15T11:25:17.128807+0000","last_fullsized":"2024-01-15T11:25:17.128807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:14.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260640+0000","last_change":"2024-01-15T11:25:16.265440+0000","last_active":"2024-01-15T11:25:17.260640+0000","last_peered":"2024-01-15T11:25:17.260640+0000","last_clean":"2024-01-15T11:25:17.260640+0000","last_became_active":"2024-01-15T11:25:16.265124+0000","last_became_peered":"2024-01-15T11:25:16.265124+0000","last_unstale":"2024-01-15T11:25:17.260640+0000","last_undegraded":"2024-01-15T11:25:17.260640+0000","last_fullsized":"2024-01-15T11:25:17.260640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:28:28.895688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.125216+0000","last_change":"2024-01-15T11:25:15.230668+0000","last_active":"2024-01-15T11:25:17.125216+0000","last_peered":"2024-01-15T11:25:17.125216+0000","last_clean":"2024-01-15T11:25:17.125216+0000","last_became_active":"2024-01-15T11:25:15.230323+0000","last_became_peered":"2024-01-15T11:25:15.230323+0000","last_unstale":"2024-01-15T11:25:17.125216+0000","last_undegraded":"2024-01-15T11:25:17.125216+0000","last_fullsized":"2024-01-15T11:25:17.125216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:25.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.280889+0000","last_change":"2024-01-15T11:25:16.267799+0000","last_active":"2024-01-15T11:25:18.280889+0000","last_peered":"2024-01-15T11:25:18.280889+0000","last_clean":"2024-01-15T11:25:18.280889+0000","last_became_active":"2024-01-15T11:25:16.266612+0000","last_became_peered":"2024-01-15T11:25:16.266612+0000","last_unstale":"2024-01-15T11:25:18.280889+0000","last_undegraded":"2024-01-15T11:25:18.280889+0000","last_fullsized":"2024-01-15T11:25:18.280889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:17:27.965755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121479+0000","last_change":"2024-01-15T11:25:15.236904+0000","last_active":"2024-01-15T11:25:17.121479+0000","last_peered":"2024-01-15T11:25:17.121479+0000","last_clean":"2024-01-15T11:25:17.121479+0000","last_became_active":"2024-01-15T11:25:15.236628+0000","last_became_peered":"2024-01-15T11:25:15.236628+0000","last_unstale":"2024-01-15T11:25:17.121479+0000","last_undegraded":"2024-01-15T11:25:17.121479+0000","last_fullsized":"2024-01-15T11:25:17.121479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:42.327446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260517+0000","last_change":"2024-01-15T11:25:16.271784+0000","last_active":"2024-01-15T11:25:17.260517+0000","last_peered":"2024-01-15T11:25:17.260517+0000","last_clean":"2024-01-15T11:25:17.260517+0000","last_became_active":"2024-01-15T11:25:16.271432+0000","last_became_peered":"2024-01-15T11:25:16.271432+0000","last_unstale":"2024-01-15T11:25:17.260517+0000","last_undegraded":"2024-01-15T11:25:17.260517+0000","last_fullsized":"2024-01-15T11:25:17.260517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:25:29.463134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118949+0000","last_change":"2024-01-15T11:25:15.238136+0000","last_active":"2024-01-15T11:25:17.118949+0000","last_peered":"2024-01-15T11:25:17.118949+0000","last_clean":"2024-01-15T11:25:17.118949+0000","last_became_active":"2024-01-15T11:25:15.237522+0000","last_became_peered":"2024-01-15T11:25:15.237522+0000","last_unstale":"2024-01-15T11:25:17.118949+0000","last_undegraded":"2024-01-15T11:25:17.118949+0000","last_fullsized":"2024-01-15T11:25:17.118949+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:16.499396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.261247+0000","last_change":"2024-01-15T11:25:16.263503+0000","last_active":"2024-01-15T11:25:17.261247+0000","last_peered":"2024-01-15T11:25:17.261247+0000","last_clean":"2024-01-15T11:25:17.261247+0000","last_became_active":"2024-01-15T11:25:16.263208+0000","last_became_peered":"2024-01-15T11:25:16.263208+0000","last_unstale":"2024-01-15T11:25:17.261247+0000","last_undegraded":"2024-01-15T11:25:17.261247+0000","last_fullsized":"2024-01-15T11:25:17.261247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:32:22.797112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.124941+0000","last_change":"2024-01-15T11:25:15.229209+0000","last_active":"2024-01-15T11:25:17.124941+0000","last_peered":"2024-01-15T11:25:17.124941+0000","last_clean":"2024-01-15T11:25:17.124941+0000","last_became_active":"2024-01-15T11:25:15.228899+0000","last_became_peered":"2024-01-15T11:25:15.228899+0000","last_unstale":"2024-01-15T11:25:17.124941+0000","last_undegraded":"2024-01-15T11:25:17.124941+0000","last_fullsized":"2024-01-15T11:25:17.124941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:56:32.006130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870812+0000","last_change":"2024-01-15T11:25:16.259973+0000","last_active":"2024-01-15T11:25:18.870812+0000","last_peered":"2024-01-15T11:25:18.870812+0000","last_clean":"2024-01-15T11:25:18.870812+0000","last_became_active":"2024-01-15T11:25:16.259589+0000","last_became_peered":"2024-01-15T11:25:16.259589+0000","last_unstale":"2024-01-15T11:25:18.870812+0000","last_undegraded":"2024-01-15T11:25:18.870812+0000","last_fullsized":"2024-01-15T11:25:18.870812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:28:28.895688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.128794+0000","last_change":"2024-01-15T11:25:15.234071+0000","last_active":"2024-01-15T11:25:17.128794+0000","last_peered":"2024-01-15T11:25:17.128794+0000","last_clean":"2024-01-15T11:25:17.128794+0000","last_became_active":"2024-01-15T11:25:15.233460+0000","last_became_peered":"2024-01-15T11:25:15.233460+0000","last_unstale":"2024-01-15T11:25:17.128794+0000","last_undegraded":"2024-01-15T11:25:17.128794+0000","last_fullsized":"2024-01-15T11:25:17.128794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:30.739157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.261956+0000","last_change":"2024-01-15T11:25:16.268001+0000","last_active":"2024-01-15T11:25:17.261956+0000","last_peered":"2024-01-15T11:25:17.261956+0000","last_clean":"2024-01-15T11:25:17.261956+0000","last_became_active":"2024-01-15T11:25:16.267277+0000","last_became_peered":"2024-01-15T11:25:16.267277+0000","last_unstale":"2024-01-15T11:25:17.261956+0000","last_undegraded":"2024-01-15T11:25:17.261956+0000","last_fullsized":"2024-01-15T11:25:17.261956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:19.633278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.255262+0000","last_change":"2024-01-15T11:25:16.258019+0000","last_active":"2024-01-15T11:25:17.255262+0000","last_peered":"2024-01-15T11:25:17.255262+0000","last_clean":"2024-01-15T11:25:17.255262+0000","last_became_active":"2024-01-15T11:25:16.257755+0000","last_became_peered":"2024-01-15T11:25:16.257755+0000","last_unstale":"2024-01-15T11:25:17.255262+0000","last_undegraded":"2024-01-15T11:25:17.255262+0000","last_fullsized":"2024-01-15T11:25:17.255262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:17:27.965755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121494+0000","last_change":"2024-01-15T11:25:15.235627+0000","last_active":"2024-01-15T11:25:17.121494+0000","last_peered":"2024-01-15T11:25:17.121494+0000","last_clean":"2024-01-15T11:25:17.121494+0000","last_became_active":"2024-01-15T11:25:15.235308+0000","last_became_peered":"2024-01-15T11:25:15.235308+0000","last_unstale":"2024-01-15T11:25:17.121494+0000","last_undegraded":"2024-01-15T11:25:17.121494+0000","last_fullsized":"2024-01-15T11:25:17.121494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:58.480006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870354+0000","last_change":"2024-01-15T11:25:16.260147+0000","last_active":"2024-01-15T11:25:18.870354+0000","last_peered":"2024-01-15T11:25:18.870354+0000","last_clean":"2024-01-15T11:25:18.870354+0000","last_became_active":"2024-01-15T11:25:16.259836+0000","last_became_peered":"2024-01-15T11:25:16.259836+0000","last_unstale":"2024-01-15T11:25:18.870354+0000","last_undegraded":"2024-01-15T11:25:18.870354+0000","last_fullsized":"2024-01-15T11:25:18.870354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:28:18.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.128819+0000","last_change":"2024-01-15T11:25:15.233565+0000","last_active":"2024-01-15T11:25:17.128819+0000","last_peered":"2024-01-15T11:25:17.128819+0000","last_clean":"2024-01-15T11:25:17.128819+0000","last_became_active":"2024-01-15T11:25:15.231919+0000","last_became_peered":"2024-01-15T11:25:15.231919+0000","last_unstale":"2024-01-15T11:25:17.128819+0000","last_undegraded":"2024-01-15T11:25:17.128819+0000","last_fullsized":"2024-01-15T11:25:17.128819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:16.243261+0000","last_change":"2024-01-15T11:25:03.506154+0000","last_active":"2024-01-15T11:25:16.243261+0000","last_peered":"2024-01-15T11:25:16.243261+0000","last_clean":"2024-01-15T11:25:16.243261+0000","last_became_active":"2024-01-15T11:25:03.504996+0000","last_became_peered":"2024-01-15T11:25:03.504996+0000","last_unstale":"2024-01-15T11:25:16.243261+0000","last_undegraded":"2024-01-15T11:25:16.243261+0000","last_fullsized":"2024-01-15T11:25:16.243261+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:02.494136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:02.494136+0000","last_clean_scrub_stamp":"2024-01-15T11:25:02.494136+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:22:33.043449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836254+0000","last_change":"2024-01-15T11:25:16.270301+0000","last_active":"2024-01-15T11:25:18.836254+0000","last_peered":"2024-01-15T11:25:18.836254+0000","last_clean":"2024-01-15T11:25:18.836254+0000","last_became_active":"2024-01-15T11:25:16.269969+0000","last_became_peered":"2024-01-15T11:25:16.269969+0000","last_unstale":"2024-01-15T11:25:18.836254+0000","last_undegraded":"2024-01-15T11:25:18.836254+0000","last_fullsized":"2024-01-15T11:25:18.836254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:54.759984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118117+0000","last_change":"2024-01-15T11:25:15.233382+0000","last_active":"2024-01-15T11:25:17.118117+0000","last_peered":"2024-01-15T11:25:17.118117+0000","last_clean":"2024-01-15T11:25:17.118117+0000","last_became_active":"2024-01-15T11:25:15.233111+0000","last_became_peered":"2024-01-15T11:25:15.233111+0000","last_unstale":"2024-01-15T11:25:17.118117+0000","last_undegraded":"2024-01-15T11:25:17.118117+0000","last_fullsized":"2024-01-15T11:25:17.118117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:50.865963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118244+0000","last_change":"2024-01-15T11:25:15.237505+0000","last_active":"2024-01-15T11:25:17.118244+0000","last_peered":"2024-01-15T11:25:17.118244+0000","last_clean":"2024-01-15T11:25:17.118244+0000","last_became_active":"2024-01-15T11:25:15.237206+0000","last_became_peered":"2024-01-15T11:25:15.237206+0000","last_unstale":"2024-01-15T11:25:17.118244+0000","last_undegraded":"2024-01-15T11:25:17.118244+0000","last_fullsized":"2024-01-15T11:25:17.118244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:14.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.255424+0000","last_change":"2024-01-15T11:25:16.254087+0000","last_active":"2024-01-15T11:25:17.255424+0000","last_peered":"2024-01-15T11:25:17.255424+0000","last_clean":"2024-01-15T11:25:17.255424+0000","last_became_active":"2024-01-15T11:25:16.253790+0000","last_became_peered":"2024-01-15T11:25:16.253790+0000","last_unstale":"2024-01-15T11:25:17.255424+0000","last_undegraded":"2024-01-15T11:25:17.255424+0000","last_fullsized":"2024-01-15T11:25:17.255424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:31.743954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.255927+0000","last_change":"2024-01-15T11:25:15.240170+0000","last_active":"2024-01-15T11:25:17.255927+0000","last_peered":"2024-01-15T11:25:17.255927+0000","last_clean":"2024-01-15T11:25:17.255927+0000","last_became_active":"2024-01-15T11:25:15.239842+0000","last_became_peered":"2024-01-15T11:25:15.239842+0000","last_unstale":"2024-01-15T11:25:17.255927+0000","last_undegraded":"2024-01-15T11:25:17.255927+0000","last_fullsized":"2024-01-15T11:25:17.255927+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:55:25.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870529+0000","last_change":"2024-01-15T11:25:16.247896+0000","last_active":"2024-01-15T11:25:18.870529+0000","last_peered":"2024-01-15T11:25:18.870529+0000","last_clean":"2024-01-15T11:25:18.870529+0000","last_became_active":"2024-01-15T11:25:16.247606+0000","last_became_peered":"2024-01-15T11:25:16.247606+0000","last_unstale":"2024-01-15T11:25:18.870529+0000","last_undegraded":"2024-01-15T11:25:18.870529+0000","last_fullsized":"2024-01-15T11:25:18.870529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:39:35.147679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.125401+0000","last_change":"2024-01-15T11:25:15.233571+0000","last_active":"2024-01-15T11:25:17.125401+0000","last_peered":"2024-01-15T11:25:17.125401+0000","last_clean":"2024-01-15T11:25:17.125401+0000","last_became_active":"2024-01-15T11:25:15.233180+0000","last_became_peered":"2024-01-15T11:25:15.233180+0000","last_unstale":"2024-01-15T11:25:17.125401+0000","last_undegraded":"2024-01-15T11:25:17.125401+0000","last_fullsized":"2024-01-15T11:25:17.125401+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:53.755186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836582+0000","last_change":"2024-01-15T11:25:16.253976+0000","last_active":"2024-01-15T11:25:18.836582+0000","last_peered":"2024-01-15T11:25:18.836582+0000","last_clean":"2024-01-15T11:25:18.836582+0000","last_became_active":"2024-01-15T11:25:16.253600+0000","last_became_peered":"2024-01-15T11:25:16.253600+0000","last_unstale":"2024-01-15T11:25:18.836582+0000","last_undegraded":"2024-01-15T11:25:18.836582+0000","last_fullsized":"2024-01-15T11:25:18.836582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:28:18.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118071+0000","last_change":"2024-01-15T11:25:15.235253+0000","last_active":"2024-01-15T11:25:17.118071+0000","last_peered":"2024-01-15T11:25:17.118071+0000","last_clean":"2024-01-15T11:25:17.118071+0000","last_became_active":"2024-01-15T11:25:15.234844+0000","last_became_peered":"2024-01-15T11:25:15.234844+0000","last_unstale":"2024-01-15T11:25:17.118071+0000","last_undegraded":"2024-01-15T11:25:17.118071+0000","last_fullsized":"2024-01-15T11:25:17.118071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:18.628480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.261714+0000","last_change":"2024-01-15T11:25:16.257934+0000","last_active":"2024-01-15T11:25:17.261714+0000","last_peered":"2024-01-15T11:25:17.261714+0000","last_clean":"2024-01-15T11:25:17.261714+0000","last_became_active":"2024-01-15T11:25:16.257566+0000","last_became_peered":"2024-01-15T11:25:16.257566+0000","last_unstale":"2024-01-15T11:25:17.261714+0000","last_undegraded":"2024-01-15T11:25:17.261714+0000","last_fullsized":"2024-01-15T11:25:17.261714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:54.555754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.124871+0000","last_change":"2024-01-15T11:25:15.237581+0000","last_active":"2024-01-15T11:25:17.124871+0000","last_peered":"2024-01-15T11:25:17.124871+0000","last_clean":"2024-01-15T11:25:17.124871+0000","last_became_active":"2024-01-15T11:25:15.237319+0000","last_became_peered":"2024-01-15T11:25:15.237319+0000","last_unstale":"2024-01-15T11:25:17.124871+0000","last_undegraded":"2024-01-15T11:25:17.124871+0000","last_fullsized":"2024-01-15T11:25:17.124871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:25.381779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870042+0000","last_change":"2024-01-15T11:25:16.256476+0000","last_active":"2024-01-15T11:25:18.870042+0000","last_peered":"2024-01-15T11:25:18.870042+0000","last_clean":"2024-01-15T11:25:18.870042+0000","last_became_active":"2024-01-15T11:25:16.256198+0000","last_became_peered":"2024-01-15T11:25:16.256198+0000","last_unstale":"2024-01-15T11:25:18.870042+0000","last_undegraded":"2024-01-15T11:25:18.870042+0000","last_fullsized":"2024-01-15T11:25:18.870042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:17:27.965755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.124914+0000","last_change":"2024-01-15T11:25:15.235154+0000","last_active":"2024-01-15T11:25:17.124914+0000","last_peered":"2024-01-15T11:25:17.124914+0000","last_clean":"2024-01-15T11:25:17.124914+0000","last_became_active":"2024-01-15T11:25:15.234870+0000","last_became_peered":"2024-01-15T11:25:15.234870+0000","last_unstale":"2024-01-15T11:25:17.124914+0000","last_undegraded":"2024-01-15T11:25:17.124914+0000","last_fullsized":"2024-01-15T11:25:17.124914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:50:53.755186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870250+0000","last_change":"2024-01-15T11:25:16.254780+0000","last_active":"2024-01-15T11:25:18.870250+0000","last_peered":"2024-01-15T11:25:18.870250+0000","last_clean":"2024-01-15T11:25:18.870250+0000","last_became_active":"2024-01-15T11:25:16.254519+0000","last_became_peered":"2024-01-15T11:25:16.254519+0000","last_unstale":"2024-01-15T11:25:18.870250+0000","last_undegraded":"2024-01-15T11:25:18.870250+0000","last_fullsized":"2024-01-15T11:25:18.870250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:15.719082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.118181+0000","last_change":"2024-01-15T11:25:15.238538+0000","last_active":"2024-01-15T11:25:17.118181+0000","last_peered":"2024-01-15T11:25:17.118181+0000","last_clean":"2024-01-15T11:25:17.118181+0000","last_became_active":"2024-01-15T11:25:15.238222+0000","last_became_peered":"2024-01-15T11:25:15.238222+0000","last_unstale":"2024-01-15T11:25:17.118181+0000","last_undegraded":"2024-01-15T11:25:17.118181+0000","last_fullsized":"2024-01-15T11:25:17.118181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:30.739157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.281109+0000","last_change":"2024-01-15T11:25:16.262608+0000","last_active":"2024-01-15T11:25:18.281109+0000","last_peered":"2024-01-15T11:25:18.281109+0000","last_clean":"2024-01-15T11:25:18.281109+0000","last_became_active":"2024-01-15T11:25:16.262318+0000","last_became_peered":"2024-01-15T11:25:16.262318+0000","last_unstale":"2024-01-15T11:25:18.281109+0000","last_undegraded":"2024-01-15T11:25:18.281109+0000","last_fullsized":"2024-01-15T11:25:18.281109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:15.719082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121516+0000","last_change":"2024-01-15T11:25:15.232704+0000","last_active":"2024-01-15T11:25:17.121516+0000","last_peered":"2024-01-15T11:25:17.121516+0000","last_clean":"2024-01-15T11:25:17.121516+0000","last_became_active":"2024-01-15T11:25:15.232191+0000","last_became_peered":"2024-01-15T11:25:15.232191+0000","last_unstale":"2024-01-15T11:25:17.121516+0000","last_undegraded":"2024-01-15T11:25:17.121516+0000","last_fullsized":"2024-01-15T11:25:17.121516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:16.499396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.260973+0000","last_change":"2024-01-15T11:25:16.260276+0000","last_active":"2024-01-15T11:25:17.260973+0000","last_peered":"2024-01-15T11:25:17.260973+0000","last_clean":"2024-01-15T11:25:17.260973+0000","last_became_active":"2024-01-15T11:25:16.259962+0000","last_became_peered":"2024-01-15T11:25:16.259962+0000","last_unstale":"2024-01-15T11:25:17.260973+0000","last_undegraded":"2024-01-15T11:25:17.260973+0000","last_fullsized":"2024-01-15T11:25:17.260973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:18:26.824601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.124968+0000","last_change":"2024-01-15T11:25:15.230446+0000","last_active":"2024-01-15T11:25:17.124968+0000","last_peered":"2024-01-15T11:25:17.124968+0000","last_clean":"2024-01-15T11:25:17.124968+0000","last_became_active":"2024-01-15T11:25:15.230176+0000","last_became_peered":"2024-01-15T11:25:15.230176+0000","last_unstale":"2024-01-15T11:25:17.124968+0000","last_undegraded":"2024-01-15T11:25:17.124968+0000","last_fullsized":"2024-01-15T11:25:17.124968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:43:32.421454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870752+0000","last_change":"2024-01-15T11:25:16.262438+0000","last_active":"2024-01-15T11:25:18.870752+0000","last_peered":"2024-01-15T11:25:18.870752+0000","last_clean":"2024-01-15T11:25:18.870752+0000","last_became_active":"2024-01-15T11:25:16.261849+0000","last_became_peered":"2024-01-15T11:25:16.261849+0000","last_unstale":"2024-01-15T11:25:18.870752+0000","last_undegraded":"2024-01-15T11:25:18.870752+0000","last_fullsized":"2024-01-15T11:25:18.870752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:25:29.463134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.121555+0000","last_change":"2024-01-15T11:25:15.235007+0000","last_active":"2024-01-15T11:25:17.121555+0000","last_peered":"2024-01-15T11:25:17.121555+0000","last_clean":"2024-01-15T11:25:17.121555+0000","last_became_active":"2024-01-15T11:25:15.234709+0000","last_became_peered":"2024-01-15T11:25:15.234709+0000","last_unstale":"2024-01-15T11:25:17.121555+0000","last_undegraded":"2024-01-15T11:25:17.121555+0000","last_fullsized":"2024-01-15T11:25:17.121555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:14.206613+0000","last_clean_scrub_stamp":"2024-01-15T11:25:14.206613+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:25.427111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.262727+0000","last_change":"2024-01-15T11:25:16.263407+0000","last_active":"2024-01-15T11:25:17.262727+0000","last_peered":"2024-01-15T11:25:17.262727+0000","last_clean":"2024-01-15T11:25:17.262727+0000","last_became_active":"2024-01-15T11:25:16.263120+0000","last_became_peered":"2024-01-15T11:25:16.263120+0000","last_unstale":"2024-01-15T11:25:17.262727+0000","last_undegraded":"2024-01-15T11:25:17.262727+0000","last_fullsized":"2024-01-15T11:25:17.262727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:46:07.070505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.280889+0000","last_change":"2024-01-15T11:25:16.258910+0000","last_active":"2024-01-15T11:25:18.280889+0000","last_peered":"2024-01-15T11:25:18.280889+0000","last_clean":"2024-01-15T11:25:18.280889+0000","last_became_active":"2024-01-15T11:25:16.258619+0000","last_became_peered":"2024-01-15T11:25:16.258619+0000","last_unstale":"2024-01-15T11:25:18.280889+0000","last_undegraded":"2024-01-15T11:25:18.280889+0000","last_fullsized":"2024-01-15T11:25:18.280889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:02:52.680406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870231+0000","last_change":"2024-01-15T11:25:16.259468+0000","last_active":"2024-01-15T11:25:18.870231+0000","last_peered":"2024-01-15T11:25:18.870231+0000","last_clean":"2024-01-15T11:25:18.870231+0000","last_became_active":"2024-01-15T11:25:16.258703+0000","last_became_peered":"2024-01-15T11:25:16.258703+0000","last_unstale":"2024-01-15T11:25:18.870231+0000","last_undegraded":"2024-01-15T11:25:18.870231+0000","last_fullsized":"2024-01-15T11:25:18.870231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:13:51.870761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870367+0000","last_change":"2024-01-15T11:25:16.266445+0000","last_active":"2024-01-15T11:25:18.870367+0000","last_peered":"2024-01-15T11:25:18.870367+0000","last_clean":"2024-01-15T11:25:18.870367+0000","last_became_active":"2024-01-15T11:25:16.265486+0000","last_became_peered":"2024-01-15T11:25:16.265486+0000","last_unstale":"2024-01-15T11:25:18.870367+0000","last_undegraded":"2024-01-15T11:25:18.870367+0000","last_fullsized":"2024-01-15T11:25:18.870367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:08:06.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836503+0000","last_change":"2024-01-15T11:25:16.259210+0000","last_active":"2024-01-15T11:25:18.836503+0000","last_peered":"2024-01-15T11:25:18.836503+0000","last_clean":"2024-01-15T11:25:18.836503+0000","last_became_active":"2024-01-15T11:25:16.258919+0000","last_became_peered":"2024-01-15T11:25:16.258919+0000","last_unstale":"2024-01-15T11:25:18.836503+0000","last_undegraded":"2024-01-15T11:25:18.836503+0000","last_fullsized":"2024-01-15T11:25:18.836503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:15.719082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870445+0000","last_change":"2024-01-15T11:25:16.264604+0000","last_active":"2024-01-15T11:25:18.870445+0000","last_peered":"2024-01-15T11:25:18.870445+0000","last_clean":"2024-01-15T11:25:18.870445+0000","last_became_active":"2024-01-15T11:25:16.263950+0000","last_became_peered":"2024-01-15T11:25:16.263950+0000","last_unstale":"2024-01-15T11:25:18.870445+0000","last_undegraded":"2024-01-15T11:25:18.870445+0000","last_fullsized":"2024-01-15T11:25:18.870445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:15:35.970176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:17.255642+0000","last_change":"2024-01-15T11:25:16.256747+0000","last_active":"2024-01-15T11:25:17.255642+0000","last_peered":"2024-01-15T11:25:17.255642+0000","last_clean":"2024-01-15T11:25:17.255642+0000","last_became_active":"2024-01-15T11:25:16.256464+0000","last_became_peered":"2024-01-15T11:25:16.256464+0000","last_unstale":"2024-01-15T11:25:17.255642+0000","last_undegraded":"2024-01-15T11:25:17.255642+0000","last_fullsized":"2024-01-15T11:25:17.255642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:28:18.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.281343+0000","last_change":"2024-01-15T11:25:16.268099+0000","last_active":"2024-01-15T11:25:18.281343+0000","last_peered":"2024-01-15T11:25:18.281343+0000","last_clean":"2024-01-15T11:25:18.281343+0000","last_became_active":"2024-01-15T11:25:16.267826+0000","last_became_peered":"2024-01-15T11:25:16.267826+0000","last_unstale":"2024-01-15T11:25:18.281343+0000","last_undegraded":"2024-01-15T11:25:18.281343+0000","last_fullsized":"2024-01-15T11:25:18.281343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:15:35.970176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870390+0000","last_change":"2024-01-15T11:25:16.263641+0000","last_active":"2024-01-15T11:25:18.870390+0000","last_peered":"2024-01-15T11:25:18.870390+0000","last_clean":"2024-01-15T11:25:18.870390+0000","last_became_active":"2024-01-15T11:25:16.263368+0000","last_became_peered":"2024-01-15T11:25:16.263368+0000","last_unstale":"2024-01-15T11:25:18.870390+0000","last_undegraded":"2024-01-15T11:25:18.870390+0000","last_fullsized":"2024-01-15T11:25:18.870390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:49.784954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870273+0000","last_change":"2024-01-15T11:25:16.262170+0000","last_active":"2024-01-15T11:25:18.870273+0000","last_peered":"2024-01-15T11:25:18.870273+0000","last_clean":"2024-01-15T11:25:18.870273+0000","last_became_active":"2024-01-15T11:25:16.261695+0000","last_became_peered":"2024-01-15T11:25:16.261695+0000","last_unstale":"2024-01-15T11:25:18.870273+0000","last_undegraded":"2024-01-15T11:25:18.870273+0000","last_fullsized":"2024-01-15T11:25:18.870273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:49.784954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870152+0000","last_change":"2024-01-15T11:25:16.265406+0000","last_active":"2024-01-15T11:25:18.870152+0000","last_peered":"2024-01-15T11:25:18.870152+0000","last_clean":"2024-01-15T11:25:18.870152+0000","last_became_active":"2024-01-15T11:25:16.265150+0000","last_became_peered":"2024-01-15T11:25:16.265150+0000","last_unstale":"2024-01-15T11:25:18.870152+0000","last_undegraded":"2024-01-15T11:25:18.870152+0000","last_fullsized":"2024-01-15T11:25:18.870152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:02:52.680406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.870317+0000","last_change":"2024-01-15T11:25:16.271625+0000","last_active":"2024-01-15T11:25:18.870317+0000","last_peered":"2024-01-15T11:25:18.870317+0000","last_clean":"2024-01-15T11:25:18.870317+0000","last_became_active":"2024-01-15T11:25:16.271361+0000","last_became_peered":"2024-01-15T11:25:16.271361+0000","last_unstale":"2024-01-15T11:25:18.870317+0000","last_undegraded":"2024-01-15T11:25:18.870317+0000","last_fullsized":"2024-01-15T11:25:18.870317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:15:35.970176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.836444+0000","last_change":"2024-01-15T11:25:16.271113+0000","last_active":"2024-01-15T11:25:18.836444+0000","last_peered":"2024-01-15T11:25:18.836444+0000","last_clean":"2024-01-15T11:25:18.836444+0000","last_became_active":"2024-01-15T11:25:16.270795+0000","last_became_peered":"2024-01-15T11:25:16.270795+0000","last_unstale":"2024-01-15T11:25:18.836444+0000","last_undegraded":"2024-01-15T11:25:18.836444+0000","last_fullsized":"2024-01-15T11:25:18.836444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:02:52.680406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T11:25:18.281234+0000","last_change":"2024-01-15T11:25:16.262518+0000","last_active":"2024-01-15T11:25:18.281234+0000","last_peered":"2024-01-15T11:25:18.281234+0000","last_clean":"2024-01-15T11:25:18.281234+0000","last_became_active":"2024-01-15T11:25:16.262049+0000","last_became_peered":"2024-01-15T11:25:16.262049+0000","last_unstale":"2024-01-15T11:25:18.281234+0000","last_undegraded":"2024-01-15T11:25:18.281234+0000","last_fullsized":"2024-01-15T11:25:18.281234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:15.211410+0000","last_clean_scrub_stamp":"2024-01-15T11:25:15.211410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:49.784954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T11:25:26.375 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-15T11:25:26.375 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-15T11:25:26.375 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-15T11:25:26.375 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-15T11:25:26.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.533+0000 7f9258281700 1 Processor -- start 2024-01-15T11:25:26.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.533+0000 7f9258281700 1 -- start start 2024-01-15T11:25:26.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f9258281700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f925015e080 0x7f92500cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f9258281700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f92500ce4d0 0x7f92500d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f9258281700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92500d0150 con 0x7f92500ce910 2024-01-15T11:25:26.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f9258281700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925015e460 con 0x7f92500ce4d0 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f9258281700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92500cee50 con 0x7f925015e080 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f925581c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f92500ce4d0 0x7f92500d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f925581c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f92500ce4d0 0x7f92500d1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37876/0 (socket says 172.21.15.113:37876) 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f925581c700 1 -- 172.21.15.113:0/1487769361 learned_addr learned my addr 172.21.15.113:0/1487769361 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f925601d700 1 --2- 172.21.15.113:0/1487769361 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f925015e080 0x7f92500cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f925581c700 1 -- 172.21.15.113:0/1487769361 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f925015e080 msgr2=0x7f92500cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f925581c700 1 --2- 172.21.15.113:0/1487769361 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f925015e080 0x7f92500cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f925581c700 1 -- 172.21.15.113:0/1487769361 >> v1:172.21.15.141:6789/0 conn(0x7f92500ce910 legacy=0x7f92500d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f925581c700 1 -- 172.21.15.113:0/1487769361 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92500cf130 con 0x7f92500ce4d0 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f925581c700 1 --2- 172.21.15.113:0/1487769361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500ce4d0 0x7f92500d1be0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f92400098f0 tx=0x7f9240009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369fb72484e35961 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.534+0000 7f925501b700 1 -- 172.21.15.113:0/1487769361 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924000b8a0 con 0x7f92500ce4d0 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.535+0000 7f925501b700 1 -- 172.21.15.113:0/1487769361 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924000ba00 con 0x7f92500ce4d0 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.535+0000 7f925501b700 1 -- 172.21.15.113:0/1487769361 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924000bd90 con 0x7f92500ce4d0 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.535+0000 7f9258281700 1 -- 172.21.15.113:0/1487769361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500ce4d0 msgr2=0x7f92500d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.535+0000 7f9258281700 1 --2- 172.21.15.113:0/1487769361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500ce4d0 0x7f92500d1be0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f92400098f0 tx=0x7f9240009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:26.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.535+0000 7f9258281700 1 -- 172.21.15.113:0/1487769361 shutdown_connections 2024-01-15T11:25:26.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.535+0000 7f9258281700 1 --2- 172.21.15.113:0/1487769361 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f925015e080 0x7f92500cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.535+0000 7f9258281700 1 --2- 172.21.15.113:0/1487769361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500ce4d0 0x7f92500d1be0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.535+0000 7f9258281700 1 -- 172.21.15.113:0/1487769361 >> 172.21.15.113:0/1487769361 conn(0x7f9250073130 msgr2=0x7f925009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:26.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.535+0000 7f9258281700 1 -- 172.21.15.113:0/1487769361 shutdown_connections 2024-01-15T11:25:26.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.535+0000 7f9258281700 1 -- 172.21.15.113:0/1487769361 wait complete. 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.537+0000 7f9258281700 1 Processor -- start 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.537+0000 7f9258281700 1 -- start start 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f9258281700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92500ce4d0 0x7f925015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f9258281700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f92500ce910 0x7f92501f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f9258281700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f925015e080 0x7f9250157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f9258281700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9250157fc0 con 0x7f925015e080 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f9258281700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92500c5dc0 con 0x7f92500ce4d0 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f9258281700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92500c6060 con 0x7f92500ce910 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f925681e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f925015e080 0x7f9250157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f925681e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f925015e080 0x7f9250157a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37886/0 (socket says 172.21.15.113:37886) 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f925681e700 1 -- 172.21.15.113:0/125939530 learned_addr learned my addr 172.21.15.113:0/125939530 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f925581c700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f92500ce910 0x7f92501f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f925601d700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92500ce4d0 0x7f925015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f925681e700 1 -- 172.21.15.113:0/125939530 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f92500ce910 msgr2=0x7f92501f3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f925681e700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f92500ce910 0x7f92501f3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f925681e700 1 -- 172.21.15.113:0/125939530 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92500ce4d0 msgr2=0x7f925015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f925681e700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92500ce4d0 0x7f925015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f925681e700 1 -- 172.21.15.113:0/125939530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9240009550 con 0x7f925015e080 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f925601d700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92500ce4d0 0x7f925015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f925681e700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f925015e080 0x7f9250157a80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f924800cbc0 tx=0x7f924800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f9246ffd700 1 -- 172.21.15.113:0/125939530 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9248003c00 con 0x7f925015e080 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f9246ffd700 1 -- 172.21.15.113:0/125939530 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9248003d60 con 0x7f925015e080 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f9258281700 1 -- 172.21.15.113:0/125939530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92500c6340 con 0x7f925015e080 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f9246ffd700 1 -- 172.21.15.113:0/125939530 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924800abf0 con 0x7f925015e080 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.538+0000 7f9258281700 1 -- 172.21.15.113:0/125939530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925014e350 con 0x7f925015e080 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.539+0000 7f9258281700 1 -- 172.21.15.113:0/125939530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925015a6d0 con 0x7f925015e080 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.540+0000 7f9246ffd700 1 -- 172.21.15.113:0/125939530 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f924800ad50 con 0x7f925015e080 2024-01-15T11:25:26.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.540+0000 7f9246ffd700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f923406b350 0x7f923406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.540+0000 7f9246ffd700 1 -- 172.21.15.113:0/125939530 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9248089cb0 con 0x7f925015e080 2024-01-15T11:25:26.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.540+0000 7f925601d700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f923406b350 0x7f923406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.540+0000 7f925601d700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f923406b350 0x7f923406d810 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f924c009820 tx=0x7f924c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:26.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.544+0000 7f9246ffd700 1 -- 172.21.15.113:0/125939530 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f925015a6d0 con 0x7f925015e080 2024-01-15T11:25:26.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.775+0000 7f9258281700 1 -- 172.21.15.113:0/125939530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f925015a6d0 con 0x7f925015e080 2024-01-15T11:25:26.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.775+0000 7f9246ffd700 1 -- 172.21.15.113:0/125939530 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f925015a6d0 con 0x7f925015e080 2024-01-15T11:25:26.774 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:26.774 INFO:teuthology.orchestra.run.smithi113.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-15T11:25:26.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.777+0000 7f9258281700 1 -- 172.21.15.113:0/125939530 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f923406b350 msgr2=0x7f923406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.777+0000 7f9258281700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f923406b350 0x7f923406d810 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f924c009820 tx=0x7f924c008040 comp rx=0 tx=0).stop 2024-01-15T11:25:26.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.777+0000 7f9258281700 1 -- 172.21.15.113:0/125939530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f925015e080 msgr2=0x7f9250157a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.777+0000 7f9258281700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f925015e080 0x7f9250157a80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f924800cbc0 tx=0x7f924800cf80 comp rx=0 tx=0).stop 2024-01-15T11:25:26.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.777+0000 7f9258281700 1 -- 172.21.15.113:0/125939530 shutdown_connections 2024-01-15T11:25:26.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.777+0000 7f9258281700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f923406b350 0x7f923406d810 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.777+0000 7f9258281700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f925015e080 0x7f9250157a80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.777+0000 7f9258281700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f92500ce4d0 0x7f925015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.777+0000 7f9258281700 1 --2- 172.21.15.113:0/125939530 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f92500ce910 0x7f92501f3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.777+0000 7f9258281700 1 -- 172.21.15.113:0/125939530 >> 172.21.15.113:0/125939530 conn(0x7f9250073130 msgr2=0x7f925015c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:26.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.777+0000 7f9258281700 1 -- 172.21.15.113:0/125939530 shutdown_connections 2024-01-15T11:25:26.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.777+0000 7f9258281700 1 -- 172.21.15.113:0/125939530 wait complete. 2024-01-15T11:25:26.788 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-15T11:25:26.788 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:26.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.910+0000 7fb4a688f700 1 Processor -- start 2024-01-15T11:25:26.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.910+0000 7fb4a688f700 1 -- start start 2024-01-15T11:25:26.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb4a688f700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb4a015b380 0x7fb4a00d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb4a688f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb4a015b760 0x7fb4a00b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb4a688f700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a015bf80 con 0x7fb4a00b3660 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb4a688f700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a00b3a90 con 0x7fb4a015b760 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb4a688f700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a015bc80 con 0x7fb4a015b380 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb49f7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb4a015b760 0x7fb4a00b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb4a4e2c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb4a00b3660 0x7fb4a00726a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56628/0 (socket says 172.21.15.113:56628) 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb4a4e2c700 1 -- 172.21.15.113:0/3090953030 learned_addr learned my addr 172.21.15.113:0/3090953030 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb49ffff700 1 --2- 172.21.15.113:0/3090953030 >> v2:172.21.15.141:3300/0 conn(0x7fb4a015b380 0x7fb4a00d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb49f7fe700 1 -- 172.21.15.113:0/3090953030 >> v2:172.21.15.141:3300/0 conn(0x7fb4a015b380 msgr2=0x7fb4a00d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb49f7fe700 1 --2- 172.21.15.113:0/3090953030 >> v2:172.21.15.141:3300/0 conn(0x7fb4a015b380 0x7fb4a00d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb49f7fe700 1 -- 172.21.15.113:0/3090953030 >> v1:172.21.15.113:6789/0 conn(0x7fb4a00b3660 legacy=0x7fb4a00726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb49f7fe700 1 -- 172.21.15.113:0/3090953030 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a00d0af0 con 0x7fb4a015b760 2024-01-15T11:25:26.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb49effd700 1 -- 172.21.15.113:0/3090953030 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065915486 0 0) 0x7fb4a015bf80 con 0x7fb4a00b3660 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb49f7fe700 1 --2- 172.21.15.113:0/3090953030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a015b760 0x7fb4a00b3120 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0072340 tx=0x7fb49000acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c0aaf2772cceb8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb49effd700 1 -- 172.21.15.113:0/3090953030 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49000cb70 con 0x7fb4a015b760 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb49effd700 1 -- 172.21.15.113:0/3090953030 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49000ccd0 con 0x7fb4a015b760 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.911+0000 7fb49effd700 1 -- 172.21.15.113:0/3090953030 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490018620 con 0x7fb4a015b760 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.912+0000 7fb4a688f700 1 -- 172.21.15.113:0/3090953030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a015b760 msgr2=0x7fb4a00b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.912+0000 7fb4a688f700 1 --2- 172.21.15.113:0/3090953030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a015b760 0x7fb4a00b3120 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0072340 tx=0x7fb49000acc0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.912+0000 7fb4a688f700 1 -- 172.21.15.113:0/3090953030 shutdown_connections 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.912+0000 7fb4a688f700 1 --2- 172.21.15.113:0/3090953030 >> v2:172.21.15.141:3300/0 conn(0x7fb4a015b380 0x7fb4a00d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.912+0000 7fb4a688f700 1 --2- 172.21.15.113:0/3090953030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a015b760 0x7fb4a00b3120 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.912+0000 7fb4a688f700 1 -- 172.21.15.113:0/3090953030 >> 172.21.15.113:0/3090953030 conn(0x7fb4a00b5640 msgr2=0x7fb4a00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.912+0000 7fb4a688f700 1 -- 172.21.15.113:0/3090953030 shutdown_connections 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.912+0000 7fb4a688f700 1 -- 172.21.15.113:0/3090953030 wait complete. 2024-01-15T11:25:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.913+0000 7fb4a688f700 1 Processor -- start 2024-01-15T11:25:26.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.913+0000 7fb4a688f700 1 -- start start 2024-01-15T11:25:26.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb4a688f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a00b3660 0x7fb4a00b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb4a688f700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb4a015b380 0x7fb4a01e2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb4a688f700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4a015b760 0x7fb4a01e31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb4a688f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a01e44d0 con 0x7fb4a00b3660 2024-01-15T11:25:26.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb4a688f700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a0150c20 con 0x7fb4a015b760 2024-01-15T11:25:26.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb4a688f700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a01e48f0 con 0x7fb4a015b380 2024-01-15T11:25:26.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb49ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a00b3660 0x7fb4a00b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb49ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a00b3660 0x7fb4a00b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37914/0 (socket says 172.21.15.113:37914) 2024-01-15T11:25:26.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb49ffff700 1 -- 172.21.15.113:0/71251471 learned_addr learned my addr 172.21.15.113:0/71251471 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb49f7fe700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb4a015b380 0x7fb4a01e2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb4a4e2c700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4a015b760 0x7fb4a01e31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb49ffff700 1 -- 172.21.15.113:0/71251471 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb4a015b380 msgr2=0x7fb4a01e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb49ffff700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb4a015b380 0x7fb4a01e2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb49ffff700 1 -- 172.21.15.113:0/71251471 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4a015b760 msgr2=0x7fb4a01e31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb49ffff700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4a015b760 0x7fb4a01e31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.914+0000 7fb49ffff700 1 -- 172.21.15.113:0/71251471 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49000a640 con 0x7fb4a00b3660 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.915+0000 7fb49f7fe700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb4a015b380 0x7fb4a01e2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.915+0000 7fb49ffff700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a00b3660 0x7fb4a00b2020 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb48800e810 tx=0x7fb48800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.915+0000 7fb49cff9700 1 -- 172.21.15.113:0/71251471 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488013610 con 0x7fb4a00b3660 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.915+0000 7fb49cff9700 1 -- 172.21.15.113:0/71251471 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb488013770 con 0x7fb4a00b3660 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.915+0000 7fb4a688f700 1 -- 172.21.15.113:0/71251471 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a01e3850 con 0x7fb4a00b3660 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.915+0000 7fb49cff9700 1 -- 172.21.15.113:0/71251471 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488013b20 con 0x7fb4a00b3660 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.915+0000 7fb4a688f700 1 -- 172.21.15.113:0/71251471 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a01e3e10 con 0x7fb4a00b3660 2024-01-15T11:25:26.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.916+0000 7fb4a688f700 1 -- 172.21.15.113:0/71251471 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a01e4740 con 0x7fb4a00b3660 2024-01-15T11:25:26.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.916+0000 7fb49cff9700 1 -- 172.21.15.113:0/71251471 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb488013c80 con 0x7fb4a00b3660 2024-01-15T11:25:26.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.916+0000 7fb49cff9700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb48006b350 0x7fb48006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:26.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.917+0000 7fb49cff9700 1 -- 172.21.15.113:0/71251471 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4880898e0 con 0x7fb4a00b3660 2024-01-15T11:25:26.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.917+0000 7fb49f7fe700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb48006b350 0x7fb48006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:26.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.917+0000 7fb49f7fe700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb48006b350 0x7fb48006d810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb4a015bb90 tx=0x7fb49001a040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:26.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:26.921+0000 7fb49cff9700 1 -- 172.21.15.113:0/71251471 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb488054760 con 0x7fb4a00b3660 2024-01-15T11:25:27.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.112+0000 7fb4a688f700 1 -- 172.21.15.113:0/71251471 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4a0067f60 con 0x7fb4a00b3660 2024-01-15T11:25:27.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.114+0000 7fb49cff9700 1 -- 172.21.15.113:0/71251471 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7fb488057d80 con 0x7fb4a00b3660 2024-01-15T11:25:27.113 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:27.113 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:27.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.116+0000 7fb4a688f700 1 -- 172.21.15.113:0/71251471 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb48006b350 msgr2=0x7fb48006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.116+0000 7fb4a688f700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb48006b350 0x7fb48006d810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb4a015bb90 tx=0x7fb49001a040 comp rx=0 tx=0).stop 2024-01-15T11:25:27.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.116+0000 7fb4a688f700 1 -- 172.21.15.113:0/71251471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a00b3660 msgr2=0x7fb4a00b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.116+0000 7fb4a688f700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a00b3660 0x7fb4a00b2020 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb48800e810 tx=0x7fb48800ebd0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.116+0000 7fb4a688f700 1 -- 172.21.15.113:0/71251471 shutdown_connections 2024-01-15T11:25:27.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.116+0000 7fb4a688f700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb48006b350 0x7fb48006d810 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.116+0000 7fb4a688f700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4a015b760 0x7fb4a01e31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.116+0000 7fb4a688f700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a00b3660 0x7fb4a00b2020 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.116+0000 7fb4a688f700 1 --2- 172.21.15.113:0/71251471 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb4a015b380 0x7fb4a01e2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.116+0000 7fb4a688f700 1 -- 172.21.15.113:0/71251471 >> 172.21.15.113:0/71251471 conn(0x7fb4a00b5640 msgr2=0x7fb4a006e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:27.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.116+0000 7fb4a688f700 1 -- 172.21.15.113:0/71251471 shutdown_connections 2024-01-15T11:25:27.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.116+0000 7fb4a688f700 1 -- 172.21.15.113:0/71251471 wait complete. 2024-01-15T11:25:27.128 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T11:25:13.398220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T11:25:14.589439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:27.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:27.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.253+0000 7feef7bf0700 1 Processor -- start 2024-01-15T11:25:27.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.253+0000 7feef7bf0700 1 -- start start 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.254+0000 7feef7bf0700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef015e160 0x7feef00af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.254+0000 7feef7bf0700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7feef00af800 0x7feef00713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.254+0000 7feef7bf0700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef006f200 con 0x7feef0071910 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.254+0000 7feef7bf0700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef0071da0 con 0x7feef00af800 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.254+0000 7feef7bf0700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef015e540 con 0x7feef015e160 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.254+0000 7feef618d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7feef0071910 0x7feef00b4050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56634/0 (socket says 172.21.15.113:56634) 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.254+0000 7feef618d700 1 -- 172.21.15.113:0/1566165422 learned_addr learned my addr 172.21.15.113:0/1566165422 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.254+0000 7feef518b700 1 --2- 172.21.15.113:0/1566165422 >> v2:172.21.15.141:3300/0 conn(0x7feef00af800 0x7feef00713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.254+0000 7feef598c700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef015e160 0x7feef00af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.254+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361986156 0 0) 0x7feef006f200 con 0x7feef0071910 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.254+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feedc0035b0 con 0x7feef0071910 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2582219364 0 0) 0x7feedc0035b0 con 0x7feef0071910 2024-01-15T11:25:27.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef015e160 msgr2=0x7feef00af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef015e160 0x7feef00af2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 >> v2:172.21.15.141:3300/0 conn(0x7feef00af800 msgr2=0x7feef00713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 --2- 172.21.15.113:0/1566165422 >> v2:172.21.15.141:3300/0 conn(0x7feef00af800 0x7feef00713d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef00afdd0 con 0x7feef0071910 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7feee80019f0 con 0x7feef0071910 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 >> v1:172.21.15.113:6789/0 conn(0x7feef0071910 legacy=0x7feef00b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feedc003a00 0x7feedc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feedc004850 0x7feedc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef015e160 0x7feedc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef0071da0 con 0x7feedc003a00 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef015e540 con 0x7feedc004850 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feedc00ba10 con 0x7feef015e160 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef598c700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feedc003a00 0x7feedc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef518b700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feedc004850 0x7feedc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef618d700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef015e160 0x7feedc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef598c700 1 -- 172.21.15.113:0/1566165422 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef015e160 msgr2=0x7feedc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef598c700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef015e160 0x7feedc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef598c700 1 -- 172.21.15.113:0/1566165422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feedc004850 msgr2=0x7feedc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef598c700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feedc004850 0x7feedc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef598c700 1 -- 172.21.15.113:0/1566165422 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc0035b0 con 0x7feedc003a00 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef618d700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef015e160 0x7feedc005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef598c700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feedc003a00 0x7feedc003e20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7feee0009b60 tx=0x7feedc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3623b3c5eb87e4e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee000e6c0 con 0x7feedc003a00 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee000e820 con 0x7feedc003a00 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.255+0000 7feef498a700 1 -- 172.21.15.113:0/1566165422 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee000eb90 con 0x7feedc003a00 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.256+0000 7feef7bf0700 1 -- 172.21.15.113:0/1566165422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feedc003a00 msgr2=0x7feedc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.256+0000 7feef7bf0700 1 --2- 172.21.15.113:0/1566165422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feedc003a00 0x7feedc003e20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7feee0009b60 tx=0x7feedc005880 comp rx=0 tx=0).stop 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.256+0000 7feef618d700 1 -- 172.21.15.113:0/1566165422 reap_dead start 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.256+0000 7feef7bf0700 1 -- 172.21.15.113:0/1566165422 shutdown_connections 2024-01-15T11:25:27.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.256+0000 7feef7bf0700 1 -- 172.21.15.113:0/1566165422 >> 172.21.15.113:0/1566165422 conn(0x7feef006aff0 msgr2=0x7feef006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:27.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.256+0000 7feef7bf0700 1 -- 172.21.15.113:0/1566165422 shutdown_connections 2024-01-15T11:25:27.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.256+0000 7feef7bf0700 1 -- 172.21.15.113:0/1566165422 wait complete. 2024-01-15T11:25:27.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.257+0000 7feef7bf0700 1 Processor -- start 2024-01-15T11:25:27.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.257+0000 7feef7bf0700 1 -- start start 2024-01-15T11:25:27.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef7bf0700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feedc003a00 0x7feef015e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef7bf0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef014f0d0 0x7feef015c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef7bf0700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feef00b2660 0x7feef00b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef7bf0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef00b15c0 con 0x7feef014f0d0 2024-01-15T11:25:27.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef7bf0700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef00b18a0 con 0x7feef00b2660 2024-01-15T11:25:27.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef7bf0700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef00b1b80 con 0x7feedc003a00 2024-01-15T11:25:27.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef518b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef014f0d0 0x7feef015c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef518b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef014f0d0 0x7feef015c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37936/0 (socket says 172.21.15.113:37936) 2024-01-15T11:25:27.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef518b700 1 -- 172.21.15.113:0/1599825838 learned_addr learned my addr 172.21.15.113:0/1599825838 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef518b700 1 -- 172.21.15.113:0/1599825838 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feedc003a00 msgr2=0x7feef015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef518b700 1 --2- 172.21.15.113:0/1599825838 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feedc003a00 0x7feef015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef518b700 1 -- 172.21.15.113:0/1599825838 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feef00b2660 msgr2=0x7feef00b2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef518b700 1 --2- 172.21.15.113:0/1599825838 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feef00b2660 0x7feef00b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef518b700 1 -- 172.21.15.113:0/1599825838 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef00b1e60 con 0x7feef014f0d0 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.258+0000 7feef518b700 1 --2- 172.21.15.113:0/1599825838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef014f0d0 0x7feef015c520 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feeec011aa0 tx=0x7feeec011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.259+0000 7feee67fc700 1 -- 172.21.15.113:0/1599825838 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec00f960 con 0x7feef014f0d0 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.259+0000 7feee67fc700 1 -- 172.21.15.113:0/1599825838 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec00fac0 con 0x7feef014f0d0 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.259+0000 7feef7bf0700 1 -- 172.21.15.113:0/1599825838 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef00b2140 con 0x7feef014f0d0 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.259+0000 7feee67fc700 1 -- 172.21.15.113:0/1599825838 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec00fe70 con 0x7feef014f0d0 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.259+0000 7feef7bf0700 1 -- 172.21.15.113:0/1599825838 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef015ca60 con 0x7feef014f0d0 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.259+0000 7feef7bf0700 1 -- 172.21.15.113:0/1599825838 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef00753f0 con 0x7feef014f0d0 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.260+0000 7feee67fc700 1 -- 172.21.15.113:0/1599825838 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feeec014440 con 0x7feef014f0d0 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.260+0000 7feee67fc700 1 --2- 172.21.15.113:0/1599825838 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7feed406b350 0x7feed406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.260+0000 7feee67fc700 1 -- 172.21.15.113:0/1599825838 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feeec08a380 con 0x7feef014f0d0 2024-01-15T11:25:27.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.260+0000 7feef598c700 1 --2- 172.21.15.113:0/1599825838 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7feed406b350 0x7feed406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.261+0000 7feef598c700 1 --2- 172.21.15.113:0/1599825838 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7feed406b350 0x7feed406d810 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feee0000c00 tx=0x7feee0013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:27.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.265+0000 7feee67fc700 1 -- 172.21.15.113:0/1599825838 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feeec055160 con 0x7feef014f0d0 2024-01-15T11:25:27.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.454+0000 7feef7bf0700 1 -- 172.21.15.113:0/1599825838 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feef00b2380 con 0x7feef014f0d0 2024-01-15T11:25:27.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.455+0000 7feee67fc700 1 -- 172.21.15.113:0/1599825838 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7feef00b2380 con 0x7feef014f0d0 2024-01-15T11:25:27.454 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.457+0000 7feef7bf0700 1 -- 172.21.15.113:0/1599825838 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7feed406b350 msgr2=0x7feed406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.457+0000 7feef7bf0700 1 --2- 172.21.15.113:0/1599825838 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7feed406b350 0x7feed406d810 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feee0000c00 tx=0x7feee0013000 comp rx=0 tx=0).stop 2024-01-15T11:25:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.457+0000 7feef7bf0700 1 -- 172.21.15.113:0/1599825838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef014f0d0 msgr2=0x7feef015c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.457+0000 7feef7bf0700 1 --2- 172.21.15.113:0/1599825838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef014f0d0 0x7feef015c520 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feeec011aa0 tx=0x7feeec011e60 comp rx=0 tx=0).stop 2024-01-15T11:25:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.457+0000 7feef7bf0700 1 -- 172.21.15.113:0/1599825838 shutdown_connections 2024-01-15T11:25:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.457+0000 7feef7bf0700 1 --2- 172.21.15.113:0/1599825838 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7feed406b350 0x7feed406d810 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.457+0000 7feef7bf0700 1 --2- 172.21.15.113:0/1599825838 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feef00b2660 0x7feef00b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.457+0000 7feef7bf0700 1 --2- 172.21.15.113:0/1599825838 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feedc003a00 0x7feef015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.457+0000 7feef7bf0700 1 --2- 172.21.15.113:0/1599825838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef014f0d0 0x7feef015c520 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.457+0000 7feef7bf0700 1 -- 172.21.15.113:0/1599825838 >> 172.21.15.113:0/1599825838 conn(0x7feef006aff0 msgr2=0x7feef006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.457+0000 7feef7bf0700 1 -- 172.21.15.113:0/1599825838 shutdown_connections 2024-01-15T11:25:27.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.457+0000 7feef7bf0700 1 -- 172.21.15.113:0/1599825838 wait complete. 2024-01-15T11:25:27.467 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T11:25:27.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.593+0000 7f972e97f700 1 Processor -- start 2024-01-15T11:25:27.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.593+0000 7f972e97f700 1 -- start start 2024-01-15T11:25:27.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.593+0000 7f972e97f700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f97280afbb0 0x7f972815c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.593+0000 7f972e97f700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f972815c5a0 0x7f97280713b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.593+0000 7f972e97f700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f972806d500 con 0x7f972815bc80 2024-01-15T11:25:27.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.593+0000 7f972e97f700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97280d0110 con 0x7f972815c5a0 2024-01-15T11:25:27.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.593+0000 7f972e97f700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972806d7a0 con 0x7f97280afbb0 2024-01-15T11:25:27.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.594+0000 7f972cf1c700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f972815c5a0 0x7f97280713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.594+0000 7f97277fe700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f97280afbb0 0x7f972815c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.594+0000 7f9727fff700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f972815bc80 0x7f97280cfa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:42006/0 (socket says 172.21.15.113:42006) 2024-01-15T11:25:27.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.594+0000 7f9727fff700 1 -- 172.21.15.113:0/87446050 learned_addr learned my addr 172.21.15.113:0/87446050 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.594+0000 7f97277fe700 1 -- 172.21.15.113:0/87446050 >> v2:172.21.15.141:3300/0 conn(0x7f972815c5a0 msgr2=0x7f97280713b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.594+0000 7f97277fe700 1 --2- 172.21.15.113:0/87446050 >> v2:172.21.15.141:3300/0 conn(0x7f972815c5a0 0x7f97280713b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.594+0000 7f97277fe700 1 -- 172.21.15.113:0/87446050 >> v1:172.21.15.141:6789/0 conn(0x7f972815bc80 legacy=0x7f97280cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.594+0000 7f97277fe700 1 -- 172.21.15.113:0/87446050 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972806da80 con 0x7f97280afbb0 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.594+0000 7f972cf1c700 1 --2- 172.21.15.113:0/87446050 >> v2:172.21.15.141:3300/0 conn(0x7f972815c5a0 0x7f97280713b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.594+0000 7f97277fe700 1 --2- 172.21.15.113:0/87446050 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f97280afbb0 0x7f972815c060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f971c009840 tx=0x7f971c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=308abe0d875c8d7e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.595+0000 7f9726ffd700 1 -- 172.21.15.113:0/87446050 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971c011830 con 0x7f97280afbb0 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.595+0000 7f9726ffd700 1 -- 172.21.15.113:0/87446050 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971c011990 con 0x7f97280afbb0 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.595+0000 7f9726ffd700 1 -- 172.21.15.113:0/87446050 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971c011d20 con 0x7f97280afbb0 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.595+0000 7f972e97f700 1 -- 172.21.15.113:0/87446050 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f97280afbb0 msgr2=0x7f972815c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.595+0000 7f972e97f700 1 --2- 172.21.15.113:0/87446050 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f97280afbb0 0x7f972815c060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f971c009840 tx=0x7f971c009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.595+0000 7f972e97f700 1 -- 172.21.15.113:0/87446050 shutdown_connections 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.595+0000 7f972e97f700 1 --2- 172.21.15.113:0/87446050 >> v2:172.21.15.141:3300/0 conn(0x7f972815c5a0 0x7f97280713b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.595+0000 7f972e97f700 1 --2- 172.21.15.113:0/87446050 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f97280afbb0 0x7f972815c060 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.595+0000 7f972e97f700 1 -- 172.21.15.113:0/87446050 >> 172.21.15.113:0/87446050 conn(0x7f9728074170 msgr2=0x7f972806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:27.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.595+0000 7f972e97f700 1 -- 172.21.15.113:0/87446050 shutdown_connections 2024-01-15T11:25:27.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.595+0000 7f972e97f700 1 -- 172.21.15.113:0/87446050 wait complete. 2024-01-15T11:25:27.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.596+0000 7f972e97f700 1 Processor -- start 2024-01-15T11:25:27.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.597+0000 7f972e97f700 1 -- start start 2024-01-15T11:25:27.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.597+0000 7f972e97f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97280afbb0 0x7f972814f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.597+0000 7f972e97f700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f972815bc80 0x7f97280ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.597+0000 7f972e97f700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f972815c5a0 0x7f97280c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.597+0000 7f972e97f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97280ae380 con 0x7f97280afbb0 2024-01-15T11:25:27.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.597+0000 7f972e97f700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97280c6ce0 con 0x7f972815c5a0 2024-01-15T11:25:27.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.597+0000 7f972e97f700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97280c63b0 con 0x7f972815bc80 2024-01-15T11:25:27.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.597+0000 7f9727fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97280afbb0 0x7f972814f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.597+0000 7f9727fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97280afbb0 0x7f972814f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37952/0 (socket says 172.21.15.113:37952) 2024-01-15T11:25:27.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.597+0000 7f9727fff700 1 -- 172.21.15.113:0/2162314988 learned_addr learned my addr 172.21.15.113:0/2162314988 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:27.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f972cf1c700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f972815c5a0 0x7f97280c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f97277fe700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f972815bc80 0x7f97280ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f9727fff700 1 -- 172.21.15.113:0/2162314988 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f972815bc80 msgr2=0x7f97280ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f9727fff700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f972815bc80 0x7f97280ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f9727fff700 1 -- 172.21.15.113:0/2162314988 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f972815c5a0 msgr2=0x7f97280c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f9727fff700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f972815c5a0 0x7f97280c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f9727fff700 1 -- 172.21.15.113:0/2162314988 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9720009a70 con 0x7f97280afbb0 2024-01-15T11:25:27.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f9727fff700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97280afbb0 0x7f972814f7c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f971800c520 tx=0x7f971800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:27.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f9724ff9700 1 -- 172.21.15.113:0/2162314988 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9718005d70 con 0x7f97280afbb0 2024-01-15T11:25:27.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f9724ff9700 1 -- 172.21.15.113:0/2162314988 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97180070d0 con 0x7f97280afbb0 2024-01-15T11:25:27.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f972e97f700 1 -- 172.21.15.113:0/2162314988 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971c0094a0 con 0x7f97280afbb0 2024-01-15T11:25:27.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f9724ff9700 1 -- 172.21.15.113:0/2162314988 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9718007440 con 0x7f97280afbb0 2024-01-15T11:25:27.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.598+0000 7f972e97f700 1 -- 172.21.15.113:0/2162314988 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97280c6970 con 0x7f97280afbb0 2024-01-15T11:25:27.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.599+0000 7f972e97f700 1 -- 172.21.15.113:0/2162314988 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97280753f0 con 0x7f97280afbb0 2024-01-15T11:25:27.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.599+0000 7f9724ff9700 1 -- 172.21.15.113:0/2162314988 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f97180075a0 con 0x7f97280afbb0 2024-01-15T11:25:27.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.600+0000 7f9724ff9700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f971406b350 0x7f971406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.600+0000 7f9724ff9700 1 -- 172.21.15.113:0/2162314988 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9718088ea0 con 0x7f97280afbb0 2024-01-15T11:25:27.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.600+0000 7f97277fe700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f971406b350 0x7f971406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.600+0000 7f97277fe700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f971406b350 0x7f971406d810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f971c000c00 tx=0x7f971c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:27.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.604+0000 7f9724ff9700 1 -- 172.21.15.113:0/2162314988 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9718053e50 con 0x7f97280afbb0 2024-01-15T11:25:27.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.793+0000 7f972e97f700 1 -- 172.21.15.113:0/2162314988 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9728069020 con 0x7f97280afbb0 2024-01-15T11:25:27.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.794+0000 7f9724ff9700 1 -- 172.21.15.113:0/2162314988 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9718017020 con 0x7f97280afbb0 2024-01-15T11:25:27.793 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-15T11:25:27.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.796+0000 7f972e97f700 1 -- 172.21.15.113:0/2162314988 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f971406b350 msgr2=0x7f971406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.796+0000 7f972e97f700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f971406b350 0x7f971406d810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f971c000c00 tx=0x7f971c008030 comp rx=0 tx=0).stop 2024-01-15T11:25:27.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.796+0000 7f972e97f700 1 -- 172.21.15.113:0/2162314988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97280afbb0 msgr2=0x7f972814f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.796+0000 7f972e97f700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97280afbb0 0x7f972814f7c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f971800c520 tx=0x7f971800c8e0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.796+0000 7f972e97f700 1 -- 172.21.15.113:0/2162314988 shutdown_connections 2024-01-15T11:25:27.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.796+0000 7f972e97f700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f971406b350 0x7f971406d810 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.796+0000 7f972e97f700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f972815c5a0 0x7f97280c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.796+0000 7f972e97f700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97280afbb0 0x7f972814f7c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.796+0000 7f972e97f700 1 --2- 172.21.15.113:0/2162314988 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f972815bc80 0x7f97280ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.796+0000 7f972e97f700 1 -- 172.21.15.113:0/2162314988 >> 172.21.15.113:0/2162314988 conn(0x7f9728074170 msgr2=0x7f972815eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:27.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.796+0000 7f972e97f700 1 -- 172.21.15.113:0/2162314988 shutdown_connections 2024-01-15T11:25:27.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.796+0000 7f972e97f700 1 -- 172.21.15.113:0/2162314988 wait complete. 2024-01-15T11:25:27.807 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T11:25:27.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.932+0000 7fd2862a6700 1 Processor -- start 2024-01-15T11:25:27.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.932+0000 7fd2862a6700 1 -- start start 2024-01-15T11:25:27.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd2862a6700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd280161690 0x7fd2800d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd2862a6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd280161a70 0x7fd2800d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd2862a6700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2800d0a70 con 0x7fd2800d1730 2024-01-15T11:25:27.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd2862a6700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2800d1c10 con 0x7fd280161690 2024-01-15T11:25:27.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd2862a6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2800b2ab0 con 0x7fd280161a70 2024-01-15T11:25:27.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd27ffff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd280161690 0x7fd2800d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd27f7fe700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd280161a70 0x7fd2800d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd284843700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fd2800d1730 0x7fd28009aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:42018/0 (socket says 172.21.15.113:42018) 2024-01-15T11:25:27.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd27ffff700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd280161690 0x7fd2800d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:58350/0 (socket says 172.21.15.113:58350) 2024-01-15T11:25:27.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd27f7fe700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd280161a70 0x7fd2800d2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:36040/0 (socket says 172.21.15.113:36040) 2024-01-15T11:25:27.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd284843700 1 -- 172.21.15.113:0/2501984863 learned_addr learned my addr 172.21.15.113:0/2501984863 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:27.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd27ffff700 1 -- 172.21.15.113:0/2501984863 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd280161a70 msgr2=0x7fd2800d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:27.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd27ffff700 1 --2- 172.21.15.113:0/2501984863 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd280161a70 0x7fd2800d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd27ffff700 1 -- 172.21.15.113:0/2501984863 >> v1:172.21.15.141:6789/0 conn(0x7fd2800d1730 legacy=0x7fd28009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd27ffff700 1 -- 172.21.15.113:0/2501984863 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2800b2d90 con 0x7fd280161690 2024-01-15T11:25:27.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd27f7fe700 1 --2- 172.21.15.113:0/2501984863 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd280161a70 0x7fd2800d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:27.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.933+0000 7fd27ffff700 1 --2- 172.21.15.113:0/2501984863 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd280161690 0x7fd2800d0f20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd270009840 tx=0x7fd270009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3268776a074e79a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:27.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.934+0000 7fd27effd700 1 -- 172.21.15.113:0/2501984863 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270004230 con 0x7fd280161690 2024-01-15T11:25:27.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.934+0000 7fd27effd700 1 -- 172.21.15.113:0/2501984863 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd270004390 con 0x7fd280161690 2024-01-15T11:25:27.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.934+0000 7fd27effd700 1 -- 172.21.15.113:0/2501984863 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2700036a0 con 0x7fd280161690 2024-01-15T11:25:27.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.934+0000 7fd2862a6700 1 -- 172.21.15.113:0/2501984863 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd280161690 msgr2=0x7fd2800d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.934+0000 7fd2862a6700 1 --2- 172.21.15.113:0/2501984863 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd280161690 0x7fd2800d0f20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd270009840 tx=0x7fd270009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:27.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.934+0000 7fd2862a6700 1 -- 172.21.15.113:0/2501984863 shutdown_connections 2024-01-15T11:25:27.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.934+0000 7fd2862a6700 1 --2- 172.21.15.113:0/2501984863 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd280161690 0x7fd2800d0f20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.934+0000 7fd2862a6700 1 --2- 172.21.15.113:0/2501984863 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd280161a70 0x7fd2800d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.934+0000 7fd2862a6700 1 -- 172.21.15.113:0/2501984863 >> 172.21.15.113:0/2501984863 conn(0x7fd2800d1e50 msgr2=0x7fd2800d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:27.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.934+0000 7fd2862a6700 1 -- 172.21.15.113:0/2501984863 shutdown_connections 2024-01-15T11:25:27.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.934+0000 7fd2862a6700 1 -- 172.21.15.113:0/2501984863 wait complete. 2024-01-15T11:25:27.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.935+0000 7fd2862a6700 1 Processor -- start 2024-01-15T11:25:27.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.936+0000 7fd2862a6700 1 -- start start 2024-01-15T11:25:27.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.936+0000 7fd2862a6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2800d1730 0x7fd2800c7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.936+0000 7fd2862a6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd280161690 0x7fd2800aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.936+0000 7fd2862a6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd280161a70 0x7fd28015b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.936+0000 7fd2862a6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2800af220 con 0x7fd280161690 2024-01-15T11:25:27.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.936+0000 7fd2862a6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2800af4c0 con 0x7fd2800d1730 2024-01-15T11:25:27.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.936+0000 7fd2862a6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2800af7a0 con 0x7fd280161a70 2024-01-15T11:25:27.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.936+0000 7fd27f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd280161690 0x7fd2800aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.936+0000 7fd27f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd280161690 0x7fd2800aece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37966/0 (socket says 172.21.15.113:37966) 2024-01-15T11:25:27.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.936+0000 7fd27f7fe700 1 -- 172.21.15.113:0/4248061630 learned_addr learned my addr 172.21.15.113:0/4248061630 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:27.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd27ffff700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2800d1730 0x7fd2800c7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd284843700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd280161a70 0x7fd28015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd27f7fe700 1 -- 172.21.15.113:0/4248061630 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd280161a70 msgr2=0x7fd28015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd27f7fe700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd280161a70 0x7fd28015b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd27f7fe700 1 -- 172.21.15.113:0/4248061630 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2800d1730 msgr2=0x7fd2800c7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:27.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd27f7fe700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2800d1730 0x7fd2800c7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:27.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd27f7fe700 1 -- 172.21.15.113:0/4248061630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2740094a0 con 0x7fd280161690 2024-01-15T11:25:27.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd27f7fe700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd280161690 0x7fd2800aece0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd27400ebe0 tx=0x7fd274009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:27.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd27cff9700 1 -- 172.21.15.113:0/4248061630 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27400c890 con 0x7fd280161690 2024-01-15T11:25:27.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd27cff9700 1 -- 172.21.15.113:0/4248061630 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27400c9f0 con 0x7fd280161690 2024-01-15T11:25:27.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd2862a6700 1 -- 172.21.15.113:0/4248061630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2700094a0 con 0x7fd280161690 2024-01-15T11:25:27.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd27cff9700 1 -- 172.21.15.113:0/4248061630 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27400cda0 con 0x7fd280161690 2024-01-15T11:25:27.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.937+0000 7fd2862a6700 1 -- 172.21.15.113:0/4248061630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2800afd60 con 0x7fd280161690 2024-01-15T11:25:27.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.938+0000 7fd2862a6700 1 -- 172.21.15.113:0/4248061630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd280069020 con 0x7fd280161690 2024-01-15T11:25:27.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.939+0000 7fd27cff9700 1 -- 172.21.15.113:0/4248061630 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd274013470 con 0x7fd280161690 2024-01-15T11:25:27.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.939+0000 7fd27cff9700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd26c06b380 0x7fd26c06d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:27.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.939+0000 7fd27cff9700 1 -- 172.21.15.113:0/4248061630 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2740899e0 con 0x7fd280161690 2024-01-15T11:25:27.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.939+0000 7fd27ffff700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd26c06b380 0x7fd26c06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:27.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.939+0000 7fd27ffff700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd26c06b380 0x7fd26c06d840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd270009840 tx=0x7fd270013040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:27.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:27.943+0000 7fd27cff9700 1 -- 172.21.15.113:0/4248061630 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd274054850 con 0x7fd280161690 2024-01-15T11:25:28.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.132+0000 7fd2862a6700 1 -- 172.21.15.113:0/4248061630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd28006c8d0 con 0x7fd280161690 2024-01-15T11:25:28.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.133+0000 7fd27cff9700 1 -- 172.21.15.113:0/4248061630 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd274057e70 con 0x7fd280161690 2024-01-15T11:25:28.132 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-15T11:25:28.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.135+0000 7fd2862a6700 1 -- 172.21.15.113:0/4248061630 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd26c06b380 msgr2=0x7fd26c06d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.135+0000 7fd2862a6700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd26c06b380 0x7fd26c06d840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd270009840 tx=0x7fd270013040 comp rx=0 tx=0).stop 2024-01-15T11:25:28.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.135+0000 7fd2862a6700 1 -- 172.21.15.113:0/4248061630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd280161690 msgr2=0x7fd2800aece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.135+0000 7fd2862a6700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd280161690 0x7fd2800aece0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd27400ebe0 tx=0x7fd274009e60 comp rx=0 tx=0).stop 2024-01-15T11:25:28.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.135+0000 7fd2862a6700 1 -- 172.21.15.113:0/4248061630 shutdown_connections 2024-01-15T11:25:28.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.135+0000 7fd2862a6700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd26c06b380 0x7fd26c06d840 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.135+0000 7fd2862a6700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd280161a70 0x7fd28015b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.135+0000 7fd2862a6700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd2800d1730 0x7fd2800c7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.135+0000 7fd2862a6700 1 --2- 172.21.15.113:0/4248061630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd280161690 0x7fd2800aece0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.135+0000 7fd2862a6700 1 -- 172.21.15.113:0/4248061630 >> 172.21.15.113:0/4248061630 conn(0x7fd2800d1e50 msgr2=0x7fd2800cd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:28.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.135+0000 7fd2862a6700 1 -- 172.21.15.113:0/4248061630 shutdown_connections 2024-01-15T11:25:28.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.135+0000 7fd2862a6700 1 -- 172.21.15.113:0/4248061630 wait complete. 2024-01-15T11:25:28.146 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T11:25:28.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.267+0000 7f98a7aa3700 1 Processor -- start 2024-01-15T11:25:28.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.268+0000 7f98a7aa3700 1 -- start start 2024-01-15T11:25:28.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.268+0000 7f98a7aa3700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f98a00d22f0 0x7f98a00d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:28.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.268+0000 7f98a7aa3700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a00d0150 con 0x7f98a00ce4d0 2024-01-15T11:25:28.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.268+0000 7f98a7aa3700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a00cee50 con 0x7f98a015e080 2024-01-15T11:25:28.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.268+0000 7f98a7aa3700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a015e460 con 0x7f98a00d22f0 2024-01-15T11:25:28.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.269+0000 7f98a503e700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f98a00ce4d0 0x7f98a00d1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56644/0 (socket says 172.21.15.113:56644) 2024-01-15T11:25:28.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.269+0000 7f98a503e700 1 -- 172.21.15.113:0/738387638 learned_addr learned my addr 172.21.15.113:0/738387638 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:28.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.269+0000 7f98a6040700 1 --2- 172.21.15.113:0/738387638 >> v2:172.21.15.113:3300/0 conn(0x7f98a00d22f0 0x7f98a00d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:28.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.269+0000 7f98a483d700 1 -- 172.21.15.113:0/738387638 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907443056 0 0) 0x7f98a00d0150 con 0x7f98a00ce4d0 2024-01-15T11:25:28.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.269+0000 7f98a483d700 1 -- 172.21.15.113:0/738387638 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f988c0035b0 con 0x7f98a00ce4d0 2024-01-15T11:25:28.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.269+0000 7f98a6040700 1 -- 172.21.15.113:0/738387638 >> v1:172.21.15.141:6789/0 conn(0x7f98a015e080 legacy=0x7f98a00cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.269+0000 7f98a6040700 1 -- 172.21.15.113:0/738387638 >> v1:172.21.15.113:6789/0 conn(0x7f98a00ce4d0 legacy=0x7f98a00d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.269+0000 7f98a6040700 1 -- 172.21.15.113:0/738387638 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a00cf130 con 0x7f98a00d22f0 2024-01-15T11:25:28.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.269+0000 7f98a6040700 1 --2- 172.21.15.113:0/738387638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a00d22f0 0x7f98a00d05e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9898009d90 tx=0x7f989800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e526537f59d1dce5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:28.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.269+0000 7f98a483d700 1 -- 172.21.15.113:0/738387638 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989800e070 con 0x7f98a00d22f0 2024-01-15T11:25:28.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.269+0000 7f98a483d700 1 -- 172.21.15.113:0/738387638 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989800bb20 con 0x7f98a00d22f0 2024-01-15T11:25:28.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.269+0000 7f98a483d700 1 -- 172.21.15.113:0/738387638 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898012430 con 0x7f98a00d22f0 2024-01-15T11:25:28.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.270+0000 7f98a7aa3700 1 -- 172.21.15.113:0/738387638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a00d22f0 msgr2=0x7f98a00d05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.270+0000 7f98a7aa3700 1 --2- 172.21.15.113:0/738387638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a00d22f0 0x7f98a00d05e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9898009d90 tx=0x7f989800b330 comp rx=0 tx=0).stop 2024-01-15T11:25:28.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.270+0000 7f98a7aa3700 1 -- 172.21.15.113:0/738387638 shutdown_connections 2024-01-15T11:25:28.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.270+0000 7f98a7aa3700 1 --2- 172.21.15.113:0/738387638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a00d22f0 0x7f98a00d05e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.270+0000 7f98a7aa3700 1 -- 172.21.15.113:0/738387638 >> 172.21.15.113:0/738387638 conn(0x7f98a0073130 msgr2=0x7f98a009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:28.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.270+0000 7f98a7aa3700 1 -- 172.21.15.113:0/738387638 shutdown_connections 2024-01-15T11:25:28.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.270+0000 7f98a7aa3700 1 -- 172.21.15.113:0/738387638 wait complete. 2024-01-15T11:25:28.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.271+0000 7f98a7aa3700 1 Processor -- start 2024-01-15T11:25:28.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.271+0000 7f98a7aa3700 1 -- start start 2024-01-15T11:25:28.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.272+0000 7f98a7aa3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a00ce4d0 0x7f98a00c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:28.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.272+0000 7f98a7aa3700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98a00d22f0 0x7f98a0156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:28.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.272+0000 7f98a7aa3700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f98a015e080 0x7f98a00b6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:28.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.272+0000 7f98a7aa3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00c6120 con 0x7f98a00ce4d0 2024-01-15T11:25:28.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.272+0000 7f98a7aa3700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a015acd0 con 0x7f98a00d22f0 2024-01-15T11:25:28.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.272+0000 7f98a7aa3700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00c6f20 con 0x7f98a015e080 2024-01-15T11:25:28.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.272+0000 7f98a583f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a00ce4d0 0x7f98a00c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:28.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.272+0000 7f98a583f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a00ce4d0 0x7f98a00c7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37986/0 (socket says 172.21.15.113:37986) 2024-01-15T11:25:28.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.272+0000 7f98a583f700 1 -- 172.21.15.113:0/3397722067 learned_addr learned my addr 172.21.15.113:0/3397722067 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:28.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98a503e700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98a00d22f0 0x7f98a0156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:28.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98a6040700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f98a015e080 0x7f98a00b6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:28.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98a583f700 1 -- 172.21.15.113:0/3397722067 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f98a015e080 msgr2=0x7f98a00b6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98a583f700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f98a015e080 0x7f98a00b6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98a583f700 1 -- 172.21.15.113:0/3397722067 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98a00d22f0 msgr2=0x7f98a0156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98a583f700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98a00d22f0 0x7f98a0156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98a583f700 1 -- 172.21.15.113:0/3397722067 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9898009a70 con 0x7f98a00ce4d0 2024-01-15T11:25:28.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98a583f700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a00ce4d0 0x7f98a00c7fe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f989000c450 tx=0x7f989000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:28.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98967fc700 1 -- 172.21.15.113:0/3397722067 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98900052d0 con 0x7f98a00ce4d0 2024-01-15T11:25:28.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98967fc700 1 -- 172.21.15.113:0/3397722067 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9890005430 con 0x7f98a00ce4d0 2024-01-15T11:25:28.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98a7aa3700 1 -- 172.21.15.113:0/3397722067 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a00c8520 con 0x7f98a00ce4d0 2024-01-15T11:25:28.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98967fc700 1 -- 172.21.15.113:0/3397722067 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98900038c0 con 0x7f98a00ce4d0 2024-01-15T11:25:28.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.273+0000 7f98a7aa3700 1 -- 172.21.15.113:0/3397722067 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a00c68d0 con 0x7f98a00ce4d0 2024-01-15T11:25:28.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.274+0000 7f98a7aa3700 1 -- 172.21.15.113:0/3397722067 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a0067f60 con 0x7f98a00ce4d0 2024-01-15T11:25:28.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.274+0000 7f98967fc700 1 -- 172.21.15.113:0/3397722067 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9890003a20 con 0x7f98a00ce4d0 2024-01-15T11:25:28.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.275+0000 7f98967fc700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f988406b350 0x7f988406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:28.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.275+0000 7f98967fc700 1 -- 172.21.15.113:0/3397722067 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9890088e60 con 0x7f98a00ce4d0 2024-01-15T11:25:28.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.275+0000 7f98a503e700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f988406b350 0x7f988406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:28.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.275+0000 7f98a503e700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f988406b350 0x7f988406d810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f98a00b3af0 tx=0x7f989c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:28.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.279+0000 7f98967fc700 1 -- 172.21.15.113:0/3397722067 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9890053d60 con 0x7f98a00ce4d0 2024-01-15T11:25:28.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.514+0000 7f98a7aa3700 1 -- 172.21.15.113:0/3397722067 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98a00d19c0 con 0x7f98a00ce4d0 2024-01-15T11:25:28.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.514+0000 7f98967fc700 1 -- 172.21.15.113:0/3397722067 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f9890057380 con 0x7f98a00ce4d0 2024-01-15T11:25:28.514 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:28.514 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3164513953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":3164513953},{"type":"v1","addr":"172.21.15.113:6835","nonce":3164513953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/891522731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":891522731},{"type":"v1","addr":"172.21.15.113:6837","nonce":891522731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2158560535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2158560535},{"type":"v1","addr":"172.21.15.141:6834","nonce":2158560535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T11:25:15.611582+0000","modified":"2024-01-15T11:25:18.718623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4300":{"gid":4300,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1012738257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1012738257},{"type":"v1","addr":"172.21.15.141:6835","nonce":1012738257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T11:25:28.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.517+0000 7f98a7aa3700 1 -- 172.21.15.113:0/3397722067 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f988406b350 msgr2=0x7f988406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.517+0000 7f98a7aa3700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f988406b350 0x7f988406d810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f98a00b3af0 tx=0x7f989c009040 comp rx=0 tx=0).stop 2024-01-15T11:25:28.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.517+0000 7f98a7aa3700 1 -- 172.21.15.113:0/3397722067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a00ce4d0 msgr2=0x7f98a00c7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.517+0000 7f98a7aa3700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a00ce4d0 0x7f98a00c7fe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f989000c450 tx=0x7f989000c810 comp rx=0 tx=0).stop 2024-01-15T11:25:28.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.517+0000 7f98a7aa3700 1 -- 172.21.15.113:0/3397722067 shutdown_connections 2024-01-15T11:25:28.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.517+0000 7f98a7aa3700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f988406b350 0x7f988406d810 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.517+0000 7f98a7aa3700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f98a015e080 0x7f98a00b6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.517+0000 7f98a7aa3700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a00ce4d0 0x7f98a00c7fe0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.517+0000 7f98a7aa3700 1 --2- 172.21.15.113:0/3397722067 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98a00d22f0 0x7f98a0156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.517+0000 7f98a7aa3700 1 -- 172.21.15.113:0/3397722067 >> 172.21.15.113:0/3397722067 conn(0x7f98a0073130 msgr2=0x7f98a00b32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:28.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.517+0000 7f98a7aa3700 1 -- 172.21.15.113:0/3397722067 shutdown_connections 2024-01-15T11:25:28.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.517+0000 7f98a7aa3700 1 -- 172.21.15.113:0/3397722067 wait complete. 2024-01-15T11:25:28.517 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-15T11:25:28.528 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:28.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.650+0000 7f956bc6b700 1 Processor -- start 2024-01-15T11:25:28.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.650+0000 7f956bc6b700 1 -- start start 2024-01-15T11:25:28.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f956bc6b700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f956415aee0 0x7f95640af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:28.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f956bc6b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f95640aff00 0x7f95640b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:28.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f956bc6b700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9564073520 con 0x7f95640b4170 2024-01-15T11:25:28.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f956bc6b700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95640ae280 con 0x7f956415aee0 2024-01-15T11:25:28.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f956bc6b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95640b4600 con 0x7f95640aff00 2024-01-15T11:25:28.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f9569a07700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f956415aee0 0x7f95640af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:28.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f9569206700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f95640aff00 0x7f95640b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:28.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f9569206700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f95640aff00 0x7f95640b3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:36066/0 (socket says 172.21.15.113:36066) 2024-01-15T11:25:28.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f956a208700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f95640b4170 0x7f95640ae6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:42036/0 (socket says 172.21.15.113:42036) 2024-01-15T11:25:28.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f9569206700 1 -- 172.21.15.113:0/3334203762 learned_addr learned my addr 172.21.15.113:0/3334203762 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:28.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f9569206700 1 -- 172.21.15.113:0/3334203762 >> v2:172.21.15.141:3300/0 conn(0x7f956415aee0 msgr2=0x7f95640af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:28.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f9569206700 1 --2- 172.21.15.113:0/3334203762 >> v2:172.21.15.141:3300/0 conn(0x7f956415aee0 0x7f95640af9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f9569206700 1 -- 172.21.15.113:0/3334203762 >> v1:172.21.15.141:6789/0 conn(0x7f95640b4170 legacy=0x7f95640ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f9569206700 1 -- 172.21.15.113:0/3334203762 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95640adbc0 con 0x7f95640aff00 2024-01-15T11:25:28.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f9569a07700 1 --2- 172.21.15.113:0/3334203762 >> v2:172.21.15.141:3300/0 conn(0x7f956415aee0 0x7f95640af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:28.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.651+0000 7f9568a05700 1 -- 172.21.15.113:0/3334203762 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020394784 0 0) 0x7f9564073520 con 0x7f95640b4170 2024-01-15T11:25:28.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.652+0000 7f9569206700 1 --2- 172.21.15.113:0/3334203762 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f95640aff00 0x7f95640b3c30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f95540098f0 tx=0x7f9554009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10bfc34b785e6928 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:28.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.652+0000 7f9568a05700 1 -- 172.21.15.113:0/3334203762 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554011830 con 0x7f95640aff00 2024-01-15T11:25:28.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.652+0000 7f9568a05700 1 -- 172.21.15.113:0/3334203762 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9554011990 con 0x7f95640aff00 2024-01-15T11:25:28.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.652+0000 7f956bc6b700 1 -- 172.21.15.113:0/3334203762 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f95640aff00 msgr2=0x7f95640b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.652+0000 7f956bc6b700 1 --2- 172.21.15.113:0/3334203762 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f95640aff00 0x7f95640b3c30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f95540098f0 tx=0x7f9554009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:28.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.652+0000 7f956bc6b700 1 -- 172.21.15.113:0/3334203762 shutdown_connections 2024-01-15T11:25:28.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.652+0000 7f956bc6b700 1 --2- 172.21.15.113:0/3334203762 >> v2:172.21.15.141:3300/0 conn(0x7f956415aee0 0x7f95640af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.652+0000 7f956bc6b700 1 --2- 172.21.15.113:0/3334203762 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f95640aff00 0x7f95640b3c30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.652+0000 7f956bc6b700 1 -- 172.21.15.113:0/3334203762 >> 172.21.15.113:0/3334203762 conn(0x7f95640b5640 msgr2=0x7f95640b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:28.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.652+0000 7f956bc6b700 1 -- 172.21.15.113:0/3334203762 shutdown_connections 2024-01-15T11:25:28.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.652+0000 7f956bc6b700 1 -- 172.21.15.113:0/3334203762 wait complete. 2024-01-15T11:25:28.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.654+0000 7f956bc6b700 1 Processor -- start 2024-01-15T11:25:28.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.654+0000 7f956bc6b700 1 -- start start 2024-01-15T11:25:28.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.654+0000 7f956bc6b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95640aff00 0x7f956414d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:28.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.654+0000 7f956bc6b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f95640b4170 0x7f956415c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:28.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.654+0000 7f956bc6b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f956415aee0 0x7f956409d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:28.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.654+0000 7f956bc6b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956415dfa0 con 0x7f95640aff00 2024-01-15T11:25:28.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.654+0000 7f956bc6b700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956415d320 con 0x7f956415aee0 2024-01-15T11:25:28.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.654+0000 7f956bc6b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956415d600 con 0x7f95640b4170 2024-01-15T11:25:28.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.654+0000 7f9569a07700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95640aff00 0x7f956414d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:28.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.654+0000 7f9569a07700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95640aff00 0x7f956414d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37996/0 (socket says 172.21.15.113:37996) 2024-01-15T11:25:28.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.654+0000 7f9569a07700 1 -- 172.21.15.113:0/1118046771 learned_addr learned my addr 172.21.15.113:0/1118046771 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:28.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f9569206700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f95640b4170 0x7f956415c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:28.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f956a208700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f956415aee0 0x7f956409d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:28.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f9569a07700 1 -- 172.21.15.113:0/1118046771 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f95640b4170 msgr2=0x7f956415c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f9569a07700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f95640b4170 0x7f956415c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f9569a07700 1 -- 172.21.15.113:0/1118046771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f956415aee0 msgr2=0x7f956409d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f9569a07700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f956415aee0 0x7f956409d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f9569a07700 1 -- 172.21.15.113:0/1118046771 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95600094a0 con 0x7f95640aff00 2024-01-15T11:25:28.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f956a208700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f956415aee0 0x7f956409d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:28.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f9569a07700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95640aff00 0x7f956414d570 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f956000ebe0 tx=0x7f956000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:28.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f955a7fc700 1 -- 172.21.15.113:0/1118046771 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560009780 con 0x7f95640aff00 2024-01-15T11:25:28.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f955a7fc700 1 -- 172.21.15.113:0/1118046771 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956000c850 con 0x7f95640aff00 2024-01-15T11:25:28.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f956bc6b700 1 -- 172.21.15.113:0/1118046771 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9554009550 con 0x7f95640aff00 2024-01-15T11:25:28.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f955a7fc700 1 -- 172.21.15.113:0/1118046771 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956000cbe0 con 0x7f95640aff00 2024-01-15T11:25:28.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.655+0000 7f956bc6b700 1 -- 172.21.15.113:0/1118046771 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956415dbc0 con 0x7f95640aff00 2024-01-15T11:25:28.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.656+0000 7f956bc6b700 1 -- 172.21.15.113:0/1118046771 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956415e210 con 0x7f95640aff00 2024-01-15T11:25:28.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.656+0000 7f955a7fc700 1 -- 172.21.15.113:0/1118046771 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f956000cd40 con 0x7f95640aff00 2024-01-15T11:25:28.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.657+0000 7f955a7fc700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f954806b350 0x7f954806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:28.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.657+0000 7f955a7fc700 1 -- 172.21.15.113:0/1118046771 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9560089ca0 con 0x7f95640aff00 2024-01-15T11:25:28.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.657+0000 7f9569206700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f954806b350 0x7f954806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:28.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.657+0000 7f9569206700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f954806b350 0x7f954806d810 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f95640b0300 tx=0x7f955401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:28.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.661+0000 7f955a7fc700 1 -- 172.21.15.113:0/1118046771 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9560054b20 con 0x7f95640aff00 2024-01-15T11:25:28.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.854+0000 7f956bc6b700 1 -- 172.21.15.113:0/1118046771 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9564067f60 con 0x7f95640aff00 2024-01-15T11:25:28.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.855+0000 7f955a7fc700 1 -- 172.21.15.113:0/1118046771 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f9560058140 con 0x7f95640aff00 2024-01-15T11:25:28.854 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:28.854 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:28.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.857+0000 7f956bc6b700 1 -- 172.21.15.113:0/1118046771 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f954806b350 msgr2=0x7f954806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.857+0000 7f956bc6b700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f954806b350 0x7f954806d810 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f95640b0300 tx=0x7f955401c040 comp rx=0 tx=0).stop 2024-01-15T11:25:28.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.857+0000 7f956bc6b700 1 -- 172.21.15.113:0/1118046771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95640aff00 msgr2=0x7f956414d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:28.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.857+0000 7f956bc6b700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95640aff00 0x7f956414d570 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f956000ebe0 tx=0x7f956000eef0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.857+0000 7f956bc6b700 1 -- 172.21.15.113:0/1118046771 shutdown_connections 2024-01-15T11:25:28.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.857+0000 7f956bc6b700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f954806b350 0x7f954806d810 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.857+0000 7f956bc6b700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f956415aee0 0x7f956409d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.857+0000 7f956bc6b700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95640aff00 0x7f956414d570 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.857+0000 7f956bc6b700 1 --2- 172.21.15.113:0/1118046771 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f95640b4170 0x7f956415c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:28.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.857+0000 7f956bc6b700 1 -- 172.21.15.113:0/1118046771 >> 172.21.15.113:0/1118046771 conn(0x7f95640b5640 msgr2=0x7f95640b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:28.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.857+0000 7f956bc6b700 1 -- 172.21.15.113:0/1118046771 shutdown_connections 2024-01-15T11:25:28.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.857+0000 7f956bc6b700 1 -- 172.21.15.113:0/1118046771 wait complete. 2024-01-15T11:25:28.868 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T11:25:13.398220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T11:25:14.589439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:28.869 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:28.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.994+0000 7efd893ab700 1 Processor -- start 2024-01-15T11:25:28.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.994+0000 7efd893ab700 1 -- start start 2024-01-15T11:25:29.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.994+0000 7efd893ab700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efd84161510 0x7efd840cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.994+0000 7efd893ab700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efd840b98e0 0x7efd840cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.994+0000 7efd893ab700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd84162af0 con 0x7efd840b59d0 2024-01-15T11:25:29.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd893ab700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd840d13e0 con 0x7efd84161510 2024-01-15T11:25:29.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd893ab700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd84161f70 con 0x7efd840b98e0 2024-01-15T11:25:29.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd82ffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efd84161510 0x7efd840cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd837fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7efd840b59d0 0x7efd840b5e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56652/0 (socket says 172.21.15.113:56652) 2024-01-15T11:25:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd837fe700 1 -- 172.21.15.113:0/4168300151 learned_addr learned my addr 172.21.15.113:0/4168300151 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd827fc700 1 --2- 172.21.15.113:0/4168300151 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efd840b98e0 0x7efd840cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd82ffd700 1 -- 172.21.15.113:0/4168300151 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efd840b98e0 msgr2=0x7efd840cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd82ffd700 1 --2- 172.21.15.113:0/4168300151 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efd840b98e0 0x7efd840cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd82ffd700 1 -- 172.21.15.113:0/4168300151 >> v1:172.21.15.113:6789/0 conn(0x7efd840b59d0 legacy=0x7efd840b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd82ffd700 1 -- 172.21.15.113:0/4168300151 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd84162250 con 0x7efd84161510 2024-01-15T11:25:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd81ffb700 1 -- 172.21.15.113:0/4168300151 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038291613 0 0) 0x7efd84162af0 con 0x7efd840b59d0 2024-01-15T11:25:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd827fc700 1 --2- 172.21.15.113:0/4168300151 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efd840b98e0 0x7efd840cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:29.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd82ffd700 1 --2- 172.21.15.113:0/4168300151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd84161510 0x7efd840cec20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efd78009840 tx=0x7efd78009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611009b0439988ae server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:29.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd81ffb700 1 -- 172.21.15.113:0/4168300151 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78011830 con 0x7efd84161510 2024-01-15T11:25:29.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd81ffb700 1 -- 172.21.15.113:0/4168300151 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd78011990 con 0x7efd84161510 2024-01-15T11:25:29.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.995+0000 7efd81ffb700 1 -- 172.21.15.113:0/4168300151 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78011d20 con 0x7efd84161510 2024-01-15T11:25:29.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.996+0000 7efd893ab700 1 -- 172.21.15.113:0/4168300151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd84161510 msgr2=0x7efd840cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.996+0000 7efd893ab700 1 --2- 172.21.15.113:0/4168300151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd84161510 0x7efd840cec20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efd78009840 tx=0x7efd78009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:29.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.996+0000 7efd893ab700 1 -- 172.21.15.113:0/4168300151 shutdown_connections 2024-01-15T11:25:29.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.996+0000 7efd893ab700 1 --2- 172.21.15.113:0/4168300151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd84161510 0x7efd840cec20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.996+0000 7efd893ab700 1 --2- 172.21.15.113:0/4168300151 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efd840b98e0 0x7efd840cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.996+0000 7efd893ab700 1 -- 172.21.15.113:0/4168300151 >> 172.21.15.113:0/4168300151 conn(0x7efd8406b410 msgr2=0x7efd8406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:29.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.996+0000 7efd893ab700 1 -- 172.21.15.113:0/4168300151 shutdown_connections 2024-01-15T11:25:29.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.996+0000 7efd893ab700 1 -- 172.21.15.113:0/4168300151 wait complete. 2024-01-15T11:25:29.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.997+0000 7efd893ab700 1 Processor -- start 2024-01-15T11:25:29.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.997+0000 7efd893ab700 1 -- start start 2024-01-15T11:25:29.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd893ab700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd840b59d0 0x7efd840c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd893ab700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efd840b98e0 0x7efd84149110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd893ab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd84161510 0x7efd840c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd893ab700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd840c8b20 con 0x7efd84161510 2024-01-15T11:25:29.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd893ab700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd840c7f10 con 0x7efd840b59d0 2024-01-15T11:25:29.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd893ab700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd840c81f0 con 0x7efd840b98e0 2024-01-15T11:25:29.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd837fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd84161510 0x7efd840c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd837fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd84161510 0x7efd840c9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38008/0 (socket says 172.21.15.113:38008) 2024-01-15T11:25:29.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd837fe700 1 -- 172.21.15.113:0/2867201391 learned_addr learned my addr 172.21.15.113:0/2867201391 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:29.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd837fe700 1 -- 172.21.15.113:0/2867201391 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efd840b98e0 msgr2=0x7efd84149110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd82ffd700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd840b59d0 0x7efd840c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd837fe700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efd840b98e0 0x7efd84149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd837fe700 1 -- 172.21.15.113:0/2867201391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd840b59d0 msgr2=0x7efd840c76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd837fe700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd840b59d0 0x7efd840c76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd837fe700 1 -- 172.21.15.113:0/2867201391 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd780094a0 con 0x7efd84161510 2024-01-15T11:25:29.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd82ffd700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd840b59d0 0x7efd840c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:29.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd837fe700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd84161510 0x7efd840c9cd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00cca0 tx=0x7efd7c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:29.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd6b7fe700 1 -- 172.21.15.113:0/2867201391 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c00cea0 con 0x7efd84161510 2024-01-15T11:25:29.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd6b7fe700 1 -- 172.21.15.113:0/2867201391 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7c00ade0 con 0x7efd84161510 2024-01-15T11:25:29.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.998+0000 7efd893ab700 1 -- 172.21.15.113:0/2867201391 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd840c84d0 con 0x7efd84161510 2024-01-15T11:25:29.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.999+0000 7efd6b7fe700 1 -- 172.21.15.113:0/2867201391 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c007310 con 0x7efd84161510 2024-01-15T11:25:29.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:28.999+0000 7efd893ab700 1 -- 172.21.15.113:0/2867201391 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd840c7c00 con 0x7efd84161510 2024-01-15T11:25:29.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.000+0000 7efd893ab700 1 -- 172.21.15.113:0/2867201391 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd84068450 con 0x7efd84161510 2024-01-15T11:25:29.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.000+0000 7efd6b7fe700 1 -- 172.21.15.113:0/2867201391 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efd7c007470 con 0x7efd84161510 2024-01-15T11:25:29.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.000+0000 7efd6b7fe700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efd7006b350 0x7efd7006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.001+0000 7efd82ffd700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efd7006b350 0x7efd7006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.001+0000 7efd6b7fe700 1 -- 172.21.15.113:0/2867201391 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd7c089b50 con 0x7efd84161510 2024-01-15T11:25:29.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.001+0000 7efd82ffd700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efd7006b350 0x7efd7006d810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efd780090b0 tx=0x7efd78008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:29.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.005+0000 7efd6b7fe700 1 -- 172.21.15.113:0/2867201391 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd7c0549c0 con 0x7efd84161510 2024-01-15T11:25:29.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.193+0000 7efd893ab700 1 -- 172.21.15.113:0/2867201391 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd84069020 con 0x7efd84161510 2024-01-15T11:25:29.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.193+0000 7efd6b7fe700 1 -- 172.21.15.113:0/2867201391 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd7c018020 con 0x7efd84161510 2024-01-15T11:25:29.193 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:29.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.196+0000 7efd893ab700 1 -- 172.21.15.113:0/2867201391 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efd7006b350 msgr2=0x7efd7006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.196+0000 7efd893ab700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efd7006b350 0x7efd7006d810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efd780090b0 tx=0x7efd78008030 comp rx=0 tx=0).stop 2024-01-15T11:25:29.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.196+0000 7efd893ab700 1 -- 172.21.15.113:0/2867201391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd84161510 msgr2=0x7efd840c9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.196+0000 7efd893ab700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd84161510 0x7efd840c9cd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00cca0 tx=0x7efd7c00a340 comp rx=0 tx=0).stop 2024-01-15T11:25:29.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.196+0000 7efd893ab700 1 -- 172.21.15.113:0/2867201391 shutdown_connections 2024-01-15T11:25:29.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.196+0000 7efd893ab700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efd7006b350 0x7efd7006d810 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.196+0000 7efd893ab700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd84161510 0x7efd840c9cd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.196+0000 7efd893ab700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd840b59d0 0x7efd840c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.196+0000 7efd893ab700 1 --2- 172.21.15.113:0/2867201391 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efd840b98e0 0x7efd84149110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.196+0000 7efd893ab700 1 -- 172.21.15.113:0/2867201391 >> 172.21.15.113:0/2867201391 conn(0x7efd8406b410 msgr2=0x7efd840d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:29.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.196+0000 7efd893ab700 1 -- 172.21.15.113:0/2867201391 shutdown_connections 2024-01-15T11:25:29.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.196+0000 7efd893ab700 1 -- 172.21.15.113:0/2867201391 wait complete. 2024-01-15T11:25:29.207 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T11:25:29.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.332+0000 7efce04b3700 1 Processor -- start 2024-01-15T11:25:29.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.332+0000 7efce04b3700 1 -- start start 2024-01-15T11:25:29.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efce04b3700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efcd815e160 0x7efcd80af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efce04b3700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7efcd8071ae0 0x7efcd806e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efce04b3700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd815e540 con 0x7efcd80af800 2024-01-15T11:25:29.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efce04b3700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd80afdd0 con 0x7efcd815e160 2024-01-15T11:25:29.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efce04b3700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd806d430 con 0x7efcd8071ae0 2024-01-15T11:25:29.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efcde24f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efcd815e160 0x7efcd80af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efcde24f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efcd815e160 0x7efcd80af2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38016/0 (socket says 172.21.15.113:38016) 2024-01-15T11:25:29.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efcde24f700 1 -- 172.21.15.113:0/4215113683 learned_addr learned my addr 172.21.15.113:0/4215113683 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:29.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efcdea50700 1 --2- 172.21.15.113:0/4215113683 >> v2:172.21.15.141:3300/0 conn(0x7efcd8071ae0 0x7efcd806e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efcde24f700 1 -- 172.21.15.113:0/4215113683 >> v2:172.21.15.141:3300/0 conn(0x7efcd8071ae0 msgr2=0x7efcd806e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:29.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efcde24f700 1 --2- 172.21.15.113:0/4215113683 >> v2:172.21.15.141:3300/0 conn(0x7efcd8071ae0 0x7efcd806e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efcde24f700 1 -- 172.21.15.113:0/4215113683 >> v1:172.21.15.141:6789/0 conn(0x7efcd80af800 legacy=0x7efcd80713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efcde24f700 1 -- 172.21.15.113:0/4215113683 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd806d710 con 0x7efcd815e160 2024-01-15T11:25:29.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efcde24f700 1 --2- 172.21.15.113:0/4215113683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcd815e160 0x7efcd80af2c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efcd80b3880 tx=0x7efcd4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27cb3c1e8083a86f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:29.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efcdd24d700 1 -- 172.21.15.113:0/4215113683 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd400b870 con 0x7efcd815e160 2024-01-15T11:25:29.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efcdd24d700 1 -- 172.21.15.113:0/4215113683 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd400b9d0 con 0x7efcd815e160 2024-01-15T11:25:29.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.333+0000 7efcdd24d700 1 -- 172.21.15.113:0/4215113683 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd400bd60 con 0x7efcd815e160 2024-01-15T11:25:29.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.334+0000 7efce04b3700 1 -- 172.21.15.113:0/4215113683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcd815e160 msgr2=0x7efcd80af2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.334+0000 7efce04b3700 1 --2- 172.21.15.113:0/4215113683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcd815e160 0x7efcd80af2c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efcd80b3880 tx=0x7efcd4009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.334+0000 7efce04b3700 1 -- 172.21.15.113:0/4215113683 shutdown_connections 2024-01-15T11:25:29.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.334+0000 7efce04b3700 1 --2- 172.21.15.113:0/4215113683 >> v2:172.21.15.141:3300/0 conn(0x7efcd8071ae0 0x7efcd806e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.334+0000 7efce04b3700 1 --2- 172.21.15.113:0/4215113683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcd815e160 0x7efcd80af2c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.334+0000 7efce04b3700 1 -- 172.21.15.113:0/4215113683 >> 172.21.15.113:0/4215113683 conn(0x7efcd806aff0 msgr2=0x7efcd806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:29.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.334+0000 7efce04b3700 1 -- 172.21.15.113:0/4215113683 shutdown_connections 2024-01-15T11:25:29.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.334+0000 7efce04b3700 1 -- 172.21.15.113:0/4215113683 wait complete. 2024-01-15T11:25:29.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.335+0000 7efce04b3700 1 Processor -- start 2024-01-15T11:25:29.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.335+0000 7efce04b3700 1 -- start start 2024-01-15T11:25:29.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efce04b3700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efcd8071ae0 0x7efcd80c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efce04b3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcd80af800 0x7efcd81f0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efce04b3700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcd815e160 0x7efcd81f1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efce04b3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd81606a0 con 0x7efcd80af800 2024-01-15T11:25:29.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efce04b3700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd8160980 con 0x7efcd8071ae0 2024-01-15T11:25:29.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efce04b3700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd8160c60 con 0x7efcd815e160 2024-01-15T11:25:29.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efcdda4e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcd80af800 0x7efcd81f0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efcdda4e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcd80af800 0x7efcd81f0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38028/0 (socket says 172.21.15.113:38028) 2024-01-15T11:25:29.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efcdda4e700 1 -- 172.21.15.113:0/3241598358 learned_addr learned my addr 172.21.15.113:0/3241598358 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:29.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efcde24f700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efcd8071ae0 0x7efcd80c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efcdea50700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcd815e160 0x7efcd81f1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efcdda4e700 1 -- 172.21.15.113:0/3241598358 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcd815e160 msgr2=0x7efcd81f1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efcdda4e700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcd815e160 0x7efcd81f1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efcdda4e700 1 -- 172.21.15.113:0/3241598358 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efcd8071ae0 msgr2=0x7efcd80c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efcdda4e700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efcd8071ae0 0x7efcd80c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efcdda4e700 1 -- 172.21.15.113:0/3241598358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd4009550 con 0x7efcd80af800 2024-01-15T11:25:29.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efcde24f700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efcd8071ae0 0x7efcd80c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:29.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.336+0000 7efcdda4e700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcd80af800 0x7efcd81f0cd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efcc800c4e0 tx=0x7efcc800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:29.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.337+0000 7efcceffd700 1 -- 172.21.15.113:0/3241598358 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc80052d0 con 0x7efcd80af800 2024-01-15T11:25:29.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.337+0000 7efcceffd700 1 -- 172.21.15.113:0/3241598358 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc8005430 con 0x7efcd80af800 2024-01-15T11:25:29.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.337+0000 7efce04b3700 1 -- 172.21.15.113:0/3241598358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd8160f40 con 0x7efcd80af800 2024-01-15T11:25:29.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.337+0000 7efcceffd700 1 -- 172.21.15.113:0/3241598358 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc80038c0 con 0x7efcd80af800 2024-01-15T11:25:29.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.337+0000 7efce04b3700 1 -- 172.21.15.113:0/3241598358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd80c5ef0 con 0x7efcd80af800 2024-01-15T11:25:29.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.338+0000 7efce04b3700 1 -- 172.21.15.113:0/3241598358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd80753f0 con 0x7efcd80af800 2024-01-15T11:25:29.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.338+0000 7efcceffd700 1 -- 172.21.15.113:0/3241598358 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efcc8003a20 con 0x7efcd80af800 2024-01-15T11:25:29.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.338+0000 7efcceffd700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efcbc06b350 0x7efcbc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.338+0000 7efcceffd700 1 -- 172.21.15.113:0/3241598358 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcc8088f00 con 0x7efcd80af800 2024-01-15T11:25:29.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.338+0000 7efcde24f700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efcbc06b350 0x7efcbc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.339+0000 7efcde24f700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efcbc06b350 0x7efcbc06d810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efcd4014040 tx=0x7efcd4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:29.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.343+0000 7efcceffd700 1 -- 172.21.15.113:0/3241598358 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efcc8053d70 con 0x7efcd80af800 2024-01-15T11:25:29.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.533+0000 7efce04b3700 1 -- 172.21.15.113:0/3241598358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efcd80c5ef0 con 0x7efcd80af800 2024-01-15T11:25:29.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.533+0000 7efcceffd700 1 -- 172.21.15.113:0/3241598358 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7efcc8017020 con 0x7efcd80af800 2024-01-15T11:25:29.532 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-15T11:25:29.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.535+0000 7efce04b3700 1 -- 172.21.15.113:0/3241598358 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efcbc06b350 msgr2=0x7efcbc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.535+0000 7efce04b3700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efcbc06b350 0x7efcbc06d810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efcd4014040 tx=0x7efcd4008030 comp rx=0 tx=0).stop 2024-01-15T11:25:29.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.535+0000 7efce04b3700 1 -- 172.21.15.113:0/3241598358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcd80af800 msgr2=0x7efcd81f0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.535+0000 7efce04b3700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcd80af800 0x7efcd81f0cd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efcc800c4e0 tx=0x7efcc800c8a0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.535+0000 7efce04b3700 1 -- 172.21.15.113:0/3241598358 shutdown_connections 2024-01-15T11:25:29.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.535+0000 7efce04b3700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efcbc06b350 0x7efcbc06d810 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.535+0000 7efce04b3700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcd815e160 0x7efcd81f1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.535+0000 7efce04b3700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efcd8071ae0 0x7efcd80c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.535+0000 7efce04b3700 1 --2- 172.21.15.113:0/3241598358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcd80af800 0x7efcd81f0cd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.535+0000 7efce04b3700 1 -- 172.21.15.113:0/3241598358 >> 172.21.15.113:0/3241598358 conn(0x7efcd806aff0 msgr2=0x7efcd80b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:29.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.535+0000 7efce04b3700 1 -- 172.21.15.113:0/3241598358 shutdown_connections 2024-01-15T11:25:29.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.535+0000 7efce04b3700 1 -- 172.21.15.113:0/3241598358 wait complete. 2024-01-15T11:25:29.546 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T11:25:29.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.670+0000 7f6ee59b5700 1 Processor -- start 2024-01-15T11:25:29.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.670+0000 7f6ee59b5700 1 -- start start 2024-01-15T11:25:29.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6ee59b5700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6ee00cf330 0x7f6ee00d2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6ee59b5700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6ee00b98e0 0x7f6ee00d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6ee59b5700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee0162af0 con 0x7f6ee0161510 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6ee59b5700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee00d13e0 con 0x7f6ee00b98e0 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6ee59b5700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee0161f70 con 0x7f6ee00cf330 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6edf7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6ee00b98e0 0x7f6ee00d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6edf7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6ee00b98e0 0x7f6ee00d2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38040/0 (socket says 172.21.15.113:38040) 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6edf7fe700 1 -- 172.21.15.113:0/4012449306 learned_addr learned my addr 172.21.15.113:0/4012449306 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6ede7fc700 1 --2- 172.21.15.113:0/4012449306 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6ee00cf330 0x7f6ee00d2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6edf7fe700 1 -- 172.21.15.113:0/4012449306 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6ee00cf330 msgr2=0x7f6ee00d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6edf7fe700 1 --2- 172.21.15.113:0/4012449306 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6ee00cf330 0x7f6ee00d2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6edf7fe700 1 -- 172.21.15.113:0/4012449306 >> v1:172.21.15.141:6789/0 conn(0x7f6ee0161510 legacy=0x7f6ee00cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6edf7fe700 1 -- 172.21.15.113:0/4012449306 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee0162250 con 0x7f6ee00b98e0 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6ede7fc700 1 --2- 172.21.15.113:0/4012449306 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6ee00cf330 0x7f6ee00d2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:29.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6edf7fe700 1 --2- 172.21.15.113:0/4012449306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ee00b98e0 0x7f6ee00d2c40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8009e50 tx=0x7f6ed800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ebe3cbdfc138c94 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:29.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.671+0000 7f6eddffb700 1 -- 172.21.15.113:0/4012449306 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed800e070 con 0x7f6ee00b98e0 2024-01-15T11:25:29.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.672+0000 7f6eddffb700 1 -- 172.21.15.113:0/4012449306 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed800b9c0 con 0x7f6ee00b98e0 2024-01-15T11:25:29.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.672+0000 7f6eddffb700 1 -- 172.21.15.113:0/4012449306 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed800bd70 con 0x7f6ee00b98e0 2024-01-15T11:25:29.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.672+0000 7f6ee59b5700 1 -- 172.21.15.113:0/4012449306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ee00b98e0 msgr2=0x7f6ee00d2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.672+0000 7f6ee59b5700 1 --2- 172.21.15.113:0/4012449306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ee00b98e0 0x7f6ee00d2c40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8009e50 tx=0x7f6ed800b330 comp rx=0 tx=0).stop 2024-01-15T11:25:29.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.672+0000 7f6ee59b5700 1 -- 172.21.15.113:0/4012449306 shutdown_connections 2024-01-15T11:25:29.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.672+0000 7f6ee59b5700 1 --2- 172.21.15.113:0/4012449306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ee00b98e0 0x7f6ee00d2c40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.672+0000 7f6ee59b5700 1 --2- 172.21.15.113:0/4012449306 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6ee00cf330 0x7f6ee00d2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.672+0000 7f6ee59b5700 1 -- 172.21.15.113:0/4012449306 >> 172.21.15.113:0/4012449306 conn(0x7f6ee006b410 msgr2=0x7f6ee006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:29.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.672+0000 7f6ee59b5700 1 -- 172.21.15.113:0/4012449306 shutdown_connections 2024-01-15T11:25:29.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.672+0000 7f6ee59b5700 1 -- 172.21.15.113:0/4012449306 wait complete. 2024-01-15T11:25:29.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.673+0000 7f6ee59b5700 1 Processor -- start 2024-01-15T11:25:29.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.673+0000 7f6ee59b5700 1 -- start start 2024-01-15T11:25:29.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6ee59b5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ee00b98e0 0x7f6ee014d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6ee59b5700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6ee00cf330 0x7f6ee015c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6ee59b5700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ee0161510 0x7f6ee015c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6ee59b5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00c8ac0 con 0x7f6ee00b98e0 2024-01-15T11:25:29.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6ee59b5700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee0152270 con 0x7f6ee0161510 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6ee59b5700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee01511d0 con 0x7f6ee00cf330 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6edeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ee00b98e0 0x7f6ee014d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6edeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ee00b98e0 0x7f6ee014d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38044/0 (socket says 172.21.15.113:38044) 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6edeffd700 1 -- 172.21.15.113:0/2477242683 learned_addr learned my addr 172.21.15.113:0/2477242683 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6ede7fc700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6ee00cf330 0x7f6ee015c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6edf7fe700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ee0161510 0x7f6ee015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6edeffd700 1 -- 172.21.15.113:0/2477242683 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6ee00cf330 msgr2=0x7f6ee015c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6edeffd700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6ee00cf330 0x7f6ee015c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6edeffd700 1 -- 172.21.15.113:0/2477242683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ee0161510 msgr2=0x7f6ee015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6edeffd700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ee0161510 0x7f6ee015c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.674+0000 7f6edeffd700 1 -- 172.21.15.113:0/2477242683 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed8009b30 con 0x7f6ee00b98e0 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.675+0000 7f6edeffd700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ee00b98e0 0x7f6ee014d6a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400c450 tx=0x7f6ed400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.675+0000 7f6ec7fff700 1 -- 172.21.15.113:0/2477242683 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed40052d0 con 0x7f6ee00b98e0 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.675+0000 7f6ec7fff700 1 -- 172.21.15.113:0/2477242683 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed4005430 con 0x7f6ee00b98e0 2024-01-15T11:25:29.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.675+0000 7f6ee59b5700 1 -- 172.21.15.113:0/2477242683 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee01514b0 con 0x7f6ee00b98e0 2024-01-15T11:25:29.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.675+0000 7f6ec7fff700 1 -- 172.21.15.113:0/2477242683 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed40038c0 con 0x7f6ee00b98e0 2024-01-15T11:25:29.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.675+0000 7f6ee59b5700 1 -- 172.21.15.113:0/2477242683 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee0151a70 con 0x7f6ee00b98e0 2024-01-15T11:25:29.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.676+0000 7f6ee59b5700 1 -- 172.21.15.113:0/2477242683 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee0068450 con 0x7f6ee00b98e0 2024-01-15T11:25:29.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.677+0000 7f6ec7fff700 1 -- 172.21.15.113:0/2477242683 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6ed4003a20 con 0x7f6ee00b98e0 2024-01-15T11:25:29.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.677+0000 7f6ec7fff700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6ecc06b350 0x7f6ecc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:29.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.677+0000 7f6ec7fff700 1 -- 172.21.15.113:0/2477242683 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ed40892d0 con 0x7f6ee00b98e0 2024-01-15T11:25:29.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.677+0000 7f6ede7fc700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6ecc06b350 0x7f6ecc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:29.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.677+0000 7f6ede7fc700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6ecc06b350 0x7f6ecc06d810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00756b0 tx=0x7f6ed0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:29.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.681+0000 7f6ec7fff700 1 -- 172.21.15.113:0/2477242683 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ed40542b0 con 0x7f6ee00b98e0 2024-01-15T11:25:29.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.871+0000 7f6ee59b5700 1 -- 172.21.15.113:0/2477242683 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6ee0069020 con 0x7f6ee00b98e0 2024-01-15T11:25:29.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.871+0000 7f6ec7fff700 1 -- 172.21.15.113:0/2477242683 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6ed40578d0 con 0x7f6ee00b98e0 2024-01-15T11:25:29.870 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-15T11:25:29.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.873+0000 7f6ee59b5700 1 -- 172.21.15.113:0/2477242683 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6ecc06b350 msgr2=0x7f6ecc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.873+0000 7f6ee59b5700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6ecc06b350 0x7f6ecc06d810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00756b0 tx=0x7f6ed0008040 comp rx=0 tx=0).stop 2024-01-15T11:25:29.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.873+0000 7f6ee59b5700 1 -- 172.21.15.113:0/2477242683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ee00b98e0 msgr2=0x7f6ee014d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:29.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.873+0000 7f6ee59b5700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ee00b98e0 0x7f6ee014d6a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400c450 tx=0x7f6ed400c810 comp rx=0 tx=0).stop 2024-01-15T11:25:29.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.873+0000 7f6ee59b5700 1 -- 172.21.15.113:0/2477242683 shutdown_connections 2024-01-15T11:25:29.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.873+0000 7f6ee59b5700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6ecc06b350 0x7f6ecc06d810 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.873+0000 7f6ee59b5700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6ee0161510 0x7f6ee015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.873+0000 7f6ee59b5700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ee00b98e0 0x7f6ee014d6a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.873+0000 7f6ee59b5700 1 --2- 172.21.15.113:0/2477242683 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6ee00cf330 0x7f6ee015c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:29.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.873+0000 7f6ee59b5700 1 -- 172.21.15.113:0/2477242683 >> 172.21.15.113:0/2477242683 conn(0x7f6ee006b410 msgr2=0x7f6ee01561e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:29.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.873+0000 7f6ee59b5700 1 -- 172.21.15.113:0/2477242683 shutdown_connections 2024-01-15T11:25:29.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:29.873+0000 7f6ee59b5700 1 -- 172.21.15.113:0/2477242683 wait complete. 2024-01-15T11:25:29.884 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T11:25:30.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.005+0000 7f077cf11700 1 Processor -- start 2024-01-15T11:25:30.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.005+0000 7f077cf11700 1 -- start start 2024-01-15T11:25:30.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.006+0000 7f077cf11700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f07781456b0 0x7f077806ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.006+0000 7f077cf11700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f077815c220 0x7f077815c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.006+0000 7f077cf11700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077806b570 con 0x7f0778073f90 2024-01-15T11:25:30.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.006+0000 7f077cf11700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f077806aff0 con 0x7f07781456b0 2024-01-15T11:25:30.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f077cf11700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0778145a90 con 0x7f077815c220 2024-01-15T11:25:30.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f077659c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f07781456b0 0x7f077806ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f077659c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f07781456b0 0x7f077806ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38052/0 (socket says 172.21.15.113:38052) 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f077659c700 1 -- 172.21.15.113:0/493498387 learned_addr learned my addr 172.21.15.113:0/493498387 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f0775d9b700 1 --2- 172.21.15.113:0/493498387 >> v2:172.21.15.141:3300/0 conn(0x7f077815c220 0x7f077815c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f077659c700 1 -- 172.21.15.113:0/493498387 >> v2:172.21.15.141:3300/0 conn(0x7f077815c220 msgr2=0x7f077815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f077659c700 1 --2- 172.21.15.113:0/493498387 >> v2:172.21.15.141:3300/0 conn(0x7f077815c220 0x7f077815c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f077659c700 1 -- 172.21.15.113:0/493498387 >> v1:172.21.15.141:6789/0 conn(0x7f0778073f90 legacy=0x7f077806dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f077659c700 1 -- 172.21.15.113:0/493498387 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077815cda0 con 0x7f07781456b0 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f077659c700 1 --2- 172.21.15.113:0/493498387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781456b0 0x7f077806ba00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0778072fe0 tx=0x7f076c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caca6cac3422ce38 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f077559a700 1 -- 172.21.15.113:0/493498387 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c00b870 con 0x7f07781456b0 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f077559a700 1 -- 172.21.15.113:0/493498387 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c00b9d0 con 0x7f07781456b0 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.007+0000 7f077559a700 1 -- 172.21.15.113:0/493498387 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c00bd60 con 0x7f07781456b0 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.008+0000 7f077cf11700 1 -- 172.21.15.113:0/493498387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781456b0 msgr2=0x7f077806ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.008+0000 7f077cf11700 1 --2- 172.21.15.113:0/493498387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781456b0 0x7f077806ba00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0778072fe0 tx=0x7f076c009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.008+0000 7f077cf11700 1 -- 172.21.15.113:0/493498387 shutdown_connections 2024-01-15T11:25:30.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.008+0000 7f077cf11700 1 --2- 172.21.15.113:0/493498387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781456b0 0x7f077806ba00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.008+0000 7f077cf11700 1 --2- 172.21.15.113:0/493498387 >> v2:172.21.15.141:3300/0 conn(0x7f077815c220 0x7f077815c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.008+0000 7f077cf11700 1 -- 172.21.15.113:0/493498387 >> 172.21.15.113:0/493498387 conn(0x7f07780b3130 msgr2=0x7f07780d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:30.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.008+0000 7f077cf11700 1 -- 172.21.15.113:0/493498387 shutdown_connections 2024-01-15T11:25:30.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.008+0000 7f077cf11700 1 -- 172.21.15.113:0/493498387 wait complete. 2024-01-15T11:25:30.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.009+0000 7f077cf11700 1 Processor -- start 2024-01-15T11:25:30.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.010+0000 7f077cf11700 1 -- start start 2024-01-15T11:25:30.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.010+0000 7f077cf11700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0778073f90 0x7f077815af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.010+0000 7f077cf11700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781456b0 0x7f077813de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.010+0000 7f077cf11700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f077815c220 0x7f0778140480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.010+0000 7f077cf11700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077815bac0 con 0x7f07781456b0 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.010+0000 7f077cf11700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077813ec60 con 0x7f077815c220 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.010+0000 7f077cf11700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07781e66c0 con 0x7f0778073f90 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0775d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781456b0 0x7f077813de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0775d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781456b0 0x7f077813de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38066/0 (socket says 172.21.15.113:38066) 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0775d9b700 1 -- 172.21.15.113:0/2970780527 learned_addr learned my addr 172.21.15.113:0/2970780527 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0776d9d700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f077815c220 0x7f0778140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0775d9b700 1 -- 172.21.15.113:0/2970780527 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0778073f90 msgr2=0x7f077815af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f077659c700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0778073f90 0x7f077815af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0775d9b700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0778073f90 0x7f077815af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0775d9b700 1 -- 172.21.15.113:0/2970780527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f077815c220 msgr2=0x7f0778140480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0775d9b700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f077815c220 0x7f0778140480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0775d9b700 1 -- 172.21.15.113:0/2970780527 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c009550 con 0x7f07781456b0 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0776d9d700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f077815c220 0x7f0778140480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0775d9b700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781456b0 0x7f077813de70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f076800e810 tx=0x7f076800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:30.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0766ffd700 1 -- 172.21.15.113:0/2970780527 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0768013610 con 0x7f07781456b0 2024-01-15T11:25:30.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0766ffd700 1 -- 172.21.15.113:0/2970780527 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0768013770 con 0x7f07781456b0 2024-01-15T11:25:30.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f077cf11700 1 -- 172.21.15.113:0/2970780527 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07781e69a0 con 0x7f07781456b0 2024-01-15T11:25:30.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f0766ffd700 1 -- 172.21.15.113:0/2970780527 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0768013b20 con 0x7f07781456b0 2024-01-15T11:25:30.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.011+0000 7f077cf11700 1 -- 172.21.15.113:0/2970780527 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077815b490 con 0x7f07781456b0 2024-01-15T11:25:30.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.012+0000 7f077cf11700 1 -- 172.21.15.113:0/2970780527 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07781e6ec0 con 0x7f07781456b0 2024-01-15T11:25:30.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.013+0000 7f0766ffd700 1 -- 172.21.15.113:0/2970780527 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0768013c80 con 0x7f07781456b0 2024-01-15T11:25:30.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.013+0000 7f0766ffd700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f075806b350 0x7f075806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.013+0000 7f0766ffd700 1 -- 172.21.15.113:0/2970780527 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0768089900 con 0x7f07781456b0 2024-01-15T11:25:30.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.013+0000 7f077659c700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f075806b350 0x7f075806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.013+0000 7f077659c700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f075806b350 0x7f075806d810 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f07780b1a60 tx=0x7f076c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:30.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.017+0000 7f0766ffd700 1 -- 172.21.15.113:0/2970780527 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f07781e6ec0 con 0x7f07781456b0 2024-01-15T11:25:30.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.254+0000 7f077cf11700 1 -- 172.21.15.113:0/2970780527 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07781e6ec0 con 0x7f07781456b0 2024-01-15T11:25:30.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.254+0000 7f0766ffd700 1 -- 172.21.15.113:0/2970780527 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f07781e6ec0 con 0x7f07781456b0 2024-01-15T11:25:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:30.253 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3164513953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":3164513953},{"type":"v1","addr":"172.21.15.113:6835","nonce":3164513953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/891522731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":891522731},{"type":"v1","addr":"172.21.15.113:6837","nonce":891522731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2158560535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2158560535},{"type":"v1","addr":"172.21.15.141:6834","nonce":2158560535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T11:25:15.611582+0000","modified":"2024-01-15T11:25:18.718623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4300":{"gid":4300,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1012738257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1012738257},{"type":"v1","addr":"172.21.15.141:6835","nonce":1012738257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T11:25:30.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.256+0000 7f077cf11700 1 -- 172.21.15.113:0/2970780527 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f075806b350 msgr2=0x7f075806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.256+0000 7f077cf11700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f075806b350 0x7f075806d810 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f07780b1a60 tx=0x7f076c008030 comp rx=0 tx=0).stop 2024-01-15T11:25:30.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.256+0000 7f077cf11700 1 -- 172.21.15.113:0/2970780527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781456b0 msgr2=0x7f077813de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.256+0000 7f077cf11700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781456b0 0x7f077813de70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f076800e810 tx=0x7f076800ebd0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.256+0000 7f077cf11700 1 -- 172.21.15.113:0/2970780527 shutdown_connections 2024-01-15T11:25:30.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.256+0000 7f077cf11700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f075806b350 0x7f075806d810 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.256+0000 7f077cf11700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f077815c220 0x7f0778140480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.256+0000 7f077cf11700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0778073f90 0x7f077815af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.256+0000 7f077cf11700 1 --2- 172.21.15.113:0/2970780527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781456b0 0x7f077813de70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.256+0000 7f077cf11700 1 -- 172.21.15.113:0/2970780527 >> 172.21.15.113:0/2970780527 conn(0x7f07780b3130 msgr2=0x7f07780b2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:30.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.256+0000 7f077cf11700 1 -- 172.21.15.113:0/2970780527 shutdown_connections 2024-01-15T11:25:30.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.256+0000 7f077cf11700 1 -- 172.21.15.113:0/2970780527 wait complete. 2024-01-15T11:25:30.256 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-15T11:25:30.267 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:30.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.390+0000 7f6cffd17700 1 Processor -- start 2024-01-15T11:25:30.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.390+0000 7f6cffd17700 1 -- start start 2024-01-15T11:25:30.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cffd17700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6cf81446a0 0x7f6cf80b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cffd17700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6cf80adf00 0x7f6cf806c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cffd17700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6cf806c540 0x7f6cf806da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cffd17700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf8144a80 con 0x7f6cf80adf00 2024-01-15T11:25:30.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cffd17700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf806c9b0 con 0x7f6cf806c540 2024-01-15T11:25:30.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cffd17700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf80af950 con 0x7f6cf81446a0 2024-01-15T11:25:30.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfd2b2700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6cf80adf00 0x7f6cf806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfd2b2700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6cf80adf00 0x7f6cf806c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38070/0 (socket says 172.21.15.113:38070) 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfd2b2700 1 -- 172.21.15.113:0/758544111 learned_addr learned my addr 172.21.15.113:0/758544111 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfdab3700 1 --2- 172.21.15.113:0/758544111 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6cf81446a0 0x7f6cf80b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfe2b4700 1 --2- 172.21.15.113:0/758544111 >> v2:172.21.15.141:3300/0 conn(0x7f6cf806c540 0x7f6cf806da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfd2b2700 1 -- 172.21.15.113:0/758544111 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6cf81446a0 msgr2=0x7f6cf80b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfd2b2700 1 --2- 172.21.15.113:0/758544111 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6cf81446a0 0x7f6cf80b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfd2b2700 1 -- 172.21.15.113:0/758544111 >> v2:172.21.15.141:3300/0 conn(0x7f6cf806c540 msgr2=0x7f6cf806da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfd2b2700 1 --2- 172.21.15.113:0/758544111 >> v2:172.21.15.141:3300/0 conn(0x7f6cf806c540 0x7f6cf806da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfd2b2700 1 -- 172.21.15.113:0/758544111 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf80afc30 con 0x7f6cf80adf00 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfe2b4700 1 --2- 172.21.15.113:0/758544111 >> v2:172.21.15.141:3300/0 conn(0x7f6cf806c540 0x7f6cf806da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfd2b2700 1 --2- 172.21.15.113:0/758544111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cf80adf00 0x7f6cf806c000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6cf806f530 tx=0x7f6ce800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4bba41143352823 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfcab1700 1 -- 172.21.15.113:0/758544111 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce800dd10 con 0x7f6cf80adf00 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfcab1700 1 -- 172.21.15.113:0/758544111 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce800de70 con 0x7f6cf80adf00 2024-01-15T11:25:30.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.391+0000 7f6cfcab1700 1 -- 172.21.15.113:0/758544111 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8010670 con 0x7f6cf80adf00 2024-01-15T11:25:30.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.392+0000 7f6cffd17700 1 -- 172.21.15.113:0/758544111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cf80adf00 msgr2=0x7f6cf806c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.392+0000 7f6cffd17700 1 --2- 172.21.15.113:0/758544111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cf80adf00 0x7f6cf806c000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6cf806f530 tx=0x7f6ce800bea0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.392+0000 7f6cffd17700 1 -- 172.21.15.113:0/758544111 shutdown_connections 2024-01-15T11:25:30.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.392+0000 7f6cffd17700 1 --2- 172.21.15.113:0/758544111 >> v2:172.21.15.141:3300/0 conn(0x7f6cf806c540 0x7f6cf806da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.392+0000 7f6cffd17700 1 --2- 172.21.15.113:0/758544111 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6cf81446a0 0x7f6cf80b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.392+0000 7f6cffd17700 1 --2- 172.21.15.113:0/758544111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cf80adf00 0x7f6cf806c000 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.392+0000 7f6cffd17700 1 -- 172.21.15.113:0/758544111 >> 172.21.15.113:0/758544111 conn(0x7f6cf8071b90 msgr2=0x7f6cf8071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:30.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.392+0000 7f6cffd17700 1 -- 172.21.15.113:0/758544111 shutdown_connections 2024-01-15T11:25:30.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.392+0000 7f6cffd17700 1 -- 172.21.15.113:0/758544111 wait complete. 2024-01-15T11:25:30.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.393+0000 7f6cffd17700 1 Processor -- start 2024-01-15T11:25:30.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.393+0000 7f6cffd17700 1 -- start start 2024-01-15T11:25:30.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cffd17700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cf806c540 0x7f6cf81492e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cffd17700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6cf80adf00 0x7f6cf814aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cffd17700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6cf81446a0 0x7f6cf809bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cffd17700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf8149820 con 0x7f6cf806c540 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cffd17700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf814a620 con 0x7f6cf81446a0 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cffd17700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf8149a10 con 0x7f6cf80adf00 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cfdab3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cf806c540 0x7f6cf81492e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cfdab3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cf806c540 0x7f6cf81492e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38076/0 (socket says 172.21.15.113:38076) 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cfdab3700 1 -- 172.21.15.113:0/3989311902 learned_addr learned my addr 172.21.15.113:0/3989311902 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cfdab3700 1 -- 172.21.15.113:0/3989311902 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6cf80adf00 msgr2=0x7f6cf814aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cfdab3700 1 --2- 172.21.15.113:0/3989311902 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6cf80adf00 0x7f6cf814aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cfdab3700 1 -- 172.21.15.113:0/3989311902 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6cf81446a0 msgr2=0x7f6cf809bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cfdab3700 1 --2- 172.21.15.113:0/3989311902 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6cf81446a0 0x7f6cf809bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cfdab3700 1 -- 172.21.15.113:0/3989311902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce800b7f0 con 0x7f6cf806c540 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cfdab3700 1 --2- 172.21.15.113:0/3989311902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cf806c540 0x7f6cf81492e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400d800 tx=0x7f6cf400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:30.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cee7fc700 1 -- 172.21.15.113:0/3989311902 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40125f0 con 0x7f6cf806c540 2024-01-15T11:25:30.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cee7fc700 1 -- 172.21.15.113:0/3989311902 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf4012750 con 0x7f6cf806c540 2024-01-15T11:25:30.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cffd17700 1 -- 172.21.15.113:0/3989311902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf8149cf0 con 0x7f6cf806c540 2024-01-15T11:25:30.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.394+0000 7f6cee7fc700 1 -- 172.21.15.113:0/3989311902 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4012b00 con 0x7f6cf806c540 2024-01-15T11:25:30.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.395+0000 7f6cffd17700 1 -- 172.21.15.113:0/3989311902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf814a2b0 con 0x7f6cf806c540 2024-01-15T11:25:30.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.395+0000 7f6cffd17700 1 -- 172.21.15.113:0/3989311902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf80753f0 con 0x7f6cf806c540 2024-01-15T11:25:30.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.396+0000 7f6cee7fc700 1 -- 172.21.15.113:0/3989311902 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6cf4012c60 con 0x7f6cf806c540 2024-01-15T11:25:30.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.396+0000 7f6cee7fc700 1 --2- 172.21.15.113:0/3989311902 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6cdc06b350 0x7f6cdc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.396+0000 7f6cfd2b2700 1 --2- 172.21.15.113:0/3989311902 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6cdc06b350 0x7f6cdc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.396+0000 7f6cee7fc700 1 -- 172.21.15.113:0/3989311902 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6cf4089f20 con 0x7f6cf806c540 2024-01-15T11:25:30.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.397+0000 7f6cfd2b2700 1 --2- 172.21.15.113:0/3989311902 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6cdc06b350 0x7f6cdc06d810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80ae2e0 tx=0x7f6ce8012000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:30.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.401+0000 7f6cee7fc700 1 -- 172.21.15.113:0/3989311902 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6cf4054e20 con 0x7f6cf806c540 2024-01-15T11:25:30.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.591+0000 7f6cffd17700 1 -- 172.21.15.113:0/3989311902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cf8067f60 con 0x7f6cf806c540 2024-01-15T11:25:30.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.592+0000 7f6cee7fc700 1 -- 172.21.15.113:0/3989311902 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f6cf4058440 con 0x7f6cf806c540 2024-01-15T11:25:30.591 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:30.591 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:30.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.595+0000 7f6cffd17700 1 -- 172.21.15.113:0/3989311902 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6cdc06b350 msgr2=0x7f6cdc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.595+0000 7f6cffd17700 1 --2- 172.21.15.113:0/3989311902 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6cdc06b350 0x7f6cdc06d810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80ae2e0 tx=0x7f6ce8012000 comp rx=0 tx=0).stop 2024-01-15T11:25:30.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.595+0000 7f6cffd17700 1 -- 172.21.15.113:0/3989311902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cf806c540 msgr2=0x7f6cf81492e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.595+0000 7f6cffd17700 1 --2- 172.21.15.113:0/3989311902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cf806c540 0x7f6cf81492e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400d800 tx=0x7f6cf400dbc0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.595+0000 7f6cffd17700 1 -- 172.21.15.113:0/3989311902 shutdown_connections 2024-01-15T11:25:30.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.595+0000 7f6cffd17700 1 --2- 172.21.15.113:0/3989311902 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6cdc06b350 0x7f6cdc06d810 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.595+0000 7f6cffd17700 1 --2- 172.21.15.113:0/3989311902 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6cf81446a0 0x7f6cf809bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.595+0000 7f6cffd17700 1 --2- 172.21.15.113:0/3989311902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cf806c540 0x7f6cf81492e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.595+0000 7f6cffd17700 1 --2- 172.21.15.113:0/3989311902 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6cf80adf00 0x7f6cf814aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.595+0000 7f6cffd17700 1 -- 172.21.15.113:0/3989311902 >> 172.21.15.113:0/3989311902 conn(0x7f6cf8071b90 msgr2=0x7f6cf815d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:30.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.595+0000 7f6cffd17700 1 -- 172.21.15.113:0/3989311902 shutdown_connections 2024-01-15T11:25:30.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.595+0000 7f6cffd17700 1 -- 172.21.15.113:0/3989311902 wait complete. 2024-01-15T11:25:30.606 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T11:25:30.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.728+0000 7f5776831700 1 Processor -- start 2024-01-15T11:25:30.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.728+0000 7f5776831700 1 -- start start 2024-01-15T11:25:30.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.728+0000 7f5776831700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f577015e080 0x7f57700b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.728+0000 7f5776831700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f57700af4d0 0x7f57700afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.728+0000 7f5776831700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f577006c000 0x7f577006c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.728+0000 7f5776831700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57700af0b0 con 0x7f577015e080 2024-01-15T11:25:30.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.728+0000 7f5776831700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f577006c9f0 con 0x7f57700af4d0 2024-01-15T11:25:30.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.728+0000 7f5776831700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57700b05a0 con 0x7f577006c000 2024-01-15T11:25:30.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.728+0000 7f576ffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f577015e080 0x7f57700b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576ffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f577015e080 0x7f57700b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38090/0 (socket says 172.21.15.113:38090) 2024-01-15T11:25:30.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576ffff700 1 -- 172.21.15.113:0/3184663678 learned_addr learned my addr 172.21.15.113:0/3184663678 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:30.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576f7fe700 1 --2- 172.21.15.113:0/3184663678 >> v2:172.21.15.141:3300/0 conn(0x7f57700af4d0 0x7f57700afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f5774dce700 1 --2- 172.21.15.113:0/3184663678 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f577006c000 0x7f577006c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576ffff700 1 -- 172.21.15.113:0/3184663678 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f577006c000 msgr2=0x7f577006c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576ffff700 1 --2- 172.21.15.113:0/3184663678 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f577006c000 0x7f577006c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576ffff700 1 -- 172.21.15.113:0/3184663678 >> v2:172.21.15.141:3300/0 conn(0x7f57700af4d0 msgr2=0x7f57700afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576ffff700 1 --2- 172.21.15.113:0/3184663678 >> v2:172.21.15.141:3300/0 conn(0x7f57700af4d0 0x7f57700afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576ffff700 1 -- 172.21.15.113:0/3184663678 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57700ae430 con 0x7f577015e080 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576f7fe700 1 --2- 172.21.15.113:0/3184663678 >> v2:172.21.15.141:3300/0 conn(0x7f57700af4d0 0x7f57700afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576ffff700 1 --2- 172.21.15.113:0/3184663678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f577015e080 0x7f57700b5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f577006e4e0 tx=0x7f5758009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45e6d177521557f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576effd700 1 -- 172.21.15.113:0/3184663678 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575800b870 con 0x7f577015e080 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576effd700 1 -- 172.21.15.113:0/3184663678 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575800b9d0 con 0x7f577015e080 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.729+0000 7f576effd700 1 -- 172.21.15.113:0/3184663678 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575800bd60 con 0x7f577015e080 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.730+0000 7f5776831700 1 -- 172.21.15.113:0/3184663678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f577015e080 msgr2=0x7f57700b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.730+0000 7f5776831700 1 --2- 172.21.15.113:0/3184663678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f577015e080 0x7f57700b5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f577006e4e0 tx=0x7f5758009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.730+0000 7f5776831700 1 -- 172.21.15.113:0/3184663678 shutdown_connections 2024-01-15T11:25:30.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.730+0000 7f5776831700 1 --2- 172.21.15.113:0/3184663678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f577015e080 0x7f57700b5050 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.730+0000 7f5776831700 1 --2- 172.21.15.113:0/3184663678 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f577006c000 0x7f577006c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.730+0000 7f5776831700 1 --2- 172.21.15.113:0/3184663678 >> v2:172.21.15.141:3300/0 conn(0x7f57700af4d0 0x7f57700afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.730+0000 7f5776831700 1 -- 172.21.15.113:0/3184663678 >> 172.21.15.113:0/3184663678 conn(0x7f5770071b90 msgr2=0x7f5770071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:30.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.730+0000 7f5776831700 1 -- 172.21.15.113:0/3184663678 shutdown_connections 2024-01-15T11:25:30.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.730+0000 7f5776831700 1 -- 172.21.15.113:0/3184663678 wait complete. 2024-01-15T11:25:30.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.731+0000 7f5776831700 1 Processor -- start 2024-01-15T11:25:30.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.731+0000 7f5776831700 1 -- start start 2024-01-15T11:25:30.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5776831700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f577006c000 0x7f57701f0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5776831700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57700af4d0 0x7f577015e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5776831700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f577015e080 0x7f5770159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5776831700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577015eeb0 con 0x7f577015e080 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5776831700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577015f190 con 0x7f57700af4d0 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5776831700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577015f470 con 0x7f577006c000 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5774dce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f577015e080 0x7f5770159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5774dce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f577015e080 0x7f5770159520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38094/0 (socket says 172.21.15.113:38094) 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5774dce700 1 -- 172.21.15.113:0/3072502964 learned_addr learned my addr 172.21.15.113:0/3072502964 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f576f7fe700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57700af4d0 0x7f577015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f576ffff700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f577006c000 0x7f57701f0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5774dce700 1 -- 172.21.15.113:0/3072502964 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f577006c000 msgr2=0x7f57701f0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5774dce700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f577006c000 0x7f57701f0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5774dce700 1 -- 172.21.15.113:0/3072502964 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57700af4d0 msgr2=0x7f577015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5774dce700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57700af4d0 0x7f577015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5774dce700 1 -- 172.21.15.113:0/3072502964 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5758009550 con 0x7f577015e080 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5774dce700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f577015e080 0x7f5770159520 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5768007e60 tx=0x7f576800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f576cff9700 1 -- 172.21.15.113:0/3072502964 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576800ede0 con 0x7f577015e080 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f576cff9700 1 -- 172.21.15.113:0/3072502964 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5768014420 con 0x7f577015e080 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f576cff9700 1 -- 172.21.15.113:0/3072502964 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57680147d0 con 0x7f577015e080 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5776831700 1 -- 172.21.15.113:0/3072502964 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577015f750 con 0x7f577015e080 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.732+0000 7f5776831700 1 -- 172.21.15.113:0/3072502964 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57701f14e0 con 0x7f577015e080 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.733+0000 7f5776831700 1 -- 172.21.15.113:0/3072502964 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57700753f0 con 0x7f577015e080 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.734+0000 7f576cff9700 1 -- 172.21.15.113:0/3072502964 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5768014930 con 0x7f577015e080 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.734+0000 7f576cff9700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f575006b350 0x7f575006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.734+0000 7f576cff9700 1 -- 172.21.15.113:0/3072502964 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f576808a230 con 0x7f577015e080 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.734+0000 7f576ffff700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f575006b350 0x7f575006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.735+0000 7f576ffff700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f575006b350 0x7f575006d810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5758014040 tx=0x7f5758008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:30.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.738+0000 7f576cff9700 1 -- 172.21.15.113:0/3072502964 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5768055130 con 0x7f577015e080 2024-01-15T11:25:30.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.966+0000 7f5776831700 1 -- 172.21.15.113:0/3072502964 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57700d09b0 con 0x7f577015e080 2024-01-15T11:25:30.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.966+0000 7f576cff9700 1 -- 172.21.15.113:0/3072502964 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f5768058750 con 0x7f577015e080 2024-01-15T11:25:30.965 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:30.965 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3164513953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":3164513953},{"type":"v1","addr":"172.21.15.113:6835","nonce":3164513953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/891522731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":891522731},{"type":"v1","addr":"172.21.15.113:6837","nonce":891522731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2158560535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2158560535},{"type":"v1","addr":"172.21.15.141:6834","nonce":2158560535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T11:25:15.611582+0000","modified":"2024-01-15T11:25:18.718623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4300":{"gid":4300,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1012738257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1012738257},{"type":"v1","addr":"172.21.15.141:6835","nonce":1012738257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T11:25:30.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.968+0000 7f5776831700 1 -- 172.21.15.113:0/3072502964 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f575006b350 msgr2=0x7f575006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.968+0000 7f5776831700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f575006b350 0x7f575006d810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5758014040 tx=0x7f5758008030 comp rx=0 tx=0).stop 2024-01-15T11:25:30.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.968+0000 7f5776831700 1 -- 172.21.15.113:0/3072502964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f577015e080 msgr2=0x7f5770159520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:30.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.968+0000 7f5776831700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f577015e080 0x7f5770159520 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5768007e60 tx=0x7f576800e3c0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.968+0000 7f5776831700 1 -- 172.21.15.113:0/3072502964 shutdown_connections 2024-01-15T11:25:30.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.968+0000 7f5776831700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f575006b350 0x7f575006d810 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.968+0000 7f5776831700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f577015e080 0x7f5770159520 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.968+0000 7f5776831700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f577006c000 0x7f57701f0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.968+0000 7f5776831700 1 --2- 172.21.15.113:0/3072502964 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f57700af4d0 0x7f577015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:30.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.968+0000 7f5776831700 1 -- 172.21.15.113:0/3072502964 >> 172.21.15.113:0/3072502964 conn(0x7f5770071b90 msgr2=0x7f57700b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:30.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.969+0000 7f5776831700 1 -- 172.21.15.113:0/3072502964 shutdown_connections 2024-01-15T11:25:30.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:30.969+0000 7f5776831700 1 -- 172.21.15.113:0/3072502964 wait complete. 2024-01-15T11:25:30.968 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-15T11:25:31.381 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T11:25:15.611582+0000', 'modified': '2024-01-15T11:25:18.718623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4300}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4300': {'gid': 4300, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.141:6835/1012738257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6833', 'nonce': 1012738257}, {'type': 'v1', 'addr': '172.21.15.141:6835', 'nonce': 1012738257}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T11:25:31.382 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T11:25:31.382 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-15T11:25:31.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.505+0000 7f077eb55700 1 Processor -- start 2024-01-15T11:25:31.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.505+0000 7f077eb55700 1 -- start start 2024-01-15T11:25:31.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.506+0000 7f077eb55700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f077815bed0 0x7f07780b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.506+0000 7f077eb55700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07780cd100 con 0x7f07780d2160 2024-01-15T11:25:31.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.506+0000 7f077eb55700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07780d09a0 con 0x7f07781603c0 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.506+0000 7f077eb55700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f077815c2b0 con 0x7f077815bed0 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.506+0000 7f077c8f1700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f077815bed0 0x7f07780b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.506+0000 7f077d0f2700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f07780d2160 0x7f07780cd590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56666/0 (socket says 172.21.15.113:56666) 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.506+0000 7f077d0f2700 1 -- 172.21.15.113:0/3026577131 learned_addr learned my addr 172.21.15.113:0/3026577131 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f077c8f1700 1 -- 172.21.15.113:0/3026577131 >> v1:172.21.15.141:6789/0 conn(0x7f07781603c0 legacy=0x7f07780d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f077c8f1700 1 -- 172.21.15.113:0/3026577131 >> v1:172.21.15.113:6789/0 conn(0x7f07780d2160 legacy=0x7f07780cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f077c8f1700 1 -- 172.21.15.113:0/3026577131 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0778160850 con 0x7f077815bed0 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f07777fe700 1 -- 172.21.15.113:0/3026577131 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841758454 0 0) 0x7f07780cd100 con 0x7f07780d2160 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f077c8f1700 1 --2- 172.21.15.113:0/3026577131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f077815bed0 0x7f07780b19f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f076c009840 tx=0x7f076c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb9f6bdd706e8d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f07777fe700 1 -- 172.21.15.113:0/3026577131 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c004230 con 0x7f077815bed0 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f07777fe700 1 -- 172.21.15.113:0/3026577131 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c004390 con 0x7f077815bed0 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f07777fe700 1 -- 172.21.15.113:0/3026577131 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c0036a0 con 0x7f077815bed0 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f077eb55700 1 -- 172.21.15.113:0/3026577131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f077815bed0 msgr2=0x7f07780b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f077eb55700 1 --2- 172.21.15.113:0/3026577131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f077815bed0 0x7f07780b19f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f076c009840 tx=0x7f076c009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:31.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f077eb55700 1 -- 172.21.15.113:0/3026577131 shutdown_connections 2024-01-15T11:25:31.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f077eb55700 1 --2- 172.21.15.113:0/3026577131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f077815bed0 0x7f07780b19f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f077eb55700 1 -- 172.21.15.113:0/3026577131 >> 172.21.15.113:0/3026577131 conn(0x7f0778073130 msgr2=0x7f077809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:31.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.507+0000 7f077eb55700 1 -- 172.21.15.113:0/3026577131 shutdown_connections 2024-01-15T11:25:31.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.508+0000 7f077eb55700 1 -- 172.21.15.113:0/3026577131 wait complete. 2024-01-15T11:25:31.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.509+0000 7f077eb55700 1 Processor -- start 2024-01-15T11:25:31.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.509+0000 7f077eb55700 1 -- start start 2024-01-15T11:25:31.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077eb55700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f07780d2160 0x7f07780c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077eb55700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f077815bed0 0x7f07780b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077eb55700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781603c0 0x7f07780b56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077eb55700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077806cce0 con 0x7f07781603c0 2024-01-15T11:25:31.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077eb55700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077806c060 con 0x7f07780d2160 2024-01-15T11:25:31.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077eb55700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077806c340 con 0x7f077815bed0 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077d0f2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781603c0 0x7f07780b56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077d0f2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781603c0 0x7f07780b56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38114/0 (socket says 172.21.15.113:38114) 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077d0f2700 1 -- 172.21.15.113:0/4044344059 learned_addr learned my addr 172.21.15.113:0/4044344059 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077c8f1700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f07780d2160 0x7f07780c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f0777fff700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f077815bed0 0x7f07780b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077d0f2700 1 -- 172.21.15.113:0/4044344059 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f077815bed0 msgr2=0x7f07780b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077d0f2700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f077815bed0 0x7f07780b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077d0f2700 1 -- 172.21.15.113:0/4044344059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f07780d2160 msgr2=0x7f07780c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077d0f2700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f07780d2160 0x7f07780c8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077d0f2700 1 -- 172.21.15.113:0/4044344059 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c0094a0 con 0x7f07781603c0 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077d0f2700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781603c0 0x7f07780b56c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f077000cca0 tx=0x7f077000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f07757fa700 1 -- 172.21.15.113:0/4044344059 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077000ab70 con 0x7f07781603c0 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f07757fa700 1 -- 172.21.15.113:0/4044344059 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077000acd0 con 0x7f07781603c0 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077eb55700 1 -- 172.21.15.113:0/4044344059 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077806c620 con 0x7f07781603c0 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f07757fa700 1 -- 172.21.15.113:0/4044344059 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0770007310 con 0x7f07781603c0 2024-01-15T11:25:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.510+0000 7f077eb55700 1 -- 172.21.15.113:0/4044344059 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07780c85a0 con 0x7f07781603c0 2024-01-15T11:25:31.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.512+0000 7f07757fa700 1 -- 172.21.15.113:0/4044344059 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0770007470 con 0x7f07781603c0 2024-01-15T11:25:31.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.512+0000 7f07757fa700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f075c06b350 0x7f075c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.512+0000 7f07757fa700 1 -- 172.21.15.113:0/4044344059 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0770089910 con 0x7f07781603c0 2024-01-15T11:25:31.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.512+0000 7f077c8f1700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f075c06b350 0x7f075c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.512+0000 7f077c8f1700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f075c06b350 0x7f075c06d810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f076c0090b0 tx=0x7f076c01c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:31.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.514+0000 7f078ad51b80 1 Processor -- start 2024-01-15T11:25:31.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.514+0000 7f078ad51b80 1 -- start start 2024-01-15T11:25:31.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f078ad51b80 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e677cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f078ad51b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 0x55c29e654190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f078ad51b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e6311d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f078ad51b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e5a4480 con 0x55c29e659410 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f078ad51b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e58a770 con 0x55c29e62edd0 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f078ad51b80 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c29e588090 con 0x55c29e5bcce0 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f0777fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 0x55c29e654190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f0777fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 0x55c29e654190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38118/0 (socket says 172.21.15.113:38118) 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f0777fff700 1 -- 172.21.15.113:0/1154481452 learned_addr learned my addr 172.21.15.113:0/1154481452 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f077d0f2700 1 --2- 172.21.15.113:0/1154481452 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e6311d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f0777fff700 1 -- 172.21.15.113:0/1154481452 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 msgr2=0x55c29e677cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f077c8f1700 1 --2- 172.21.15.113:0/1154481452 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e677cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f0777fff700 1 --2- 172.21.15.113:0/1154481452 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e677cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f0777fff700 1 -- 172.21.15.113:0/1154481452 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 msgr2=0x55c29e6311d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f0777fff700 1 --2- 172.21.15.113:0/1154481452 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e6311d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f0777fff700 1 -- 172.21.15.113:0/1154481452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c29e3acb10 con 0x55c29e659410 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f0777fff700 1 --2- 172.21.15.113:0/1154481452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 0x55c29e654190 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0764009a40 tx=0x7f0764009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7519d8a213554530 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f07627fc700 1 -- 172.21.15.113:0/1154481452 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07640094e0 con 0x55c29e659410 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f07627fc700 1 -- 172.21.15.113:0/1154481452 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0764004020 con 0x55c29e659410 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.515+0000 7f07627fc700 1 -- 172.21.15.113:0/1154481452 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07640043b0 con 0x55c29e659410 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.516+0000 7f078ad51b80 1 -- 172.21.15.113:0/1154481452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 msgr2=0x55c29e654190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.516+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1154481452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 0x55c29e654190 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0764009a40 tx=0x7f0764009d50 comp rx=0 tx=0).stop 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.516+0000 7f078ad51b80 1 -- 172.21.15.113:0/1154481452 shutdown_connections 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.516+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1154481452 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e6311d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.516+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1154481452 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e677cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.516+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1154481452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 0x55c29e654190 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.516+0000 7f078ad51b80 1 -- 172.21.15.113:0/1154481452 >> 172.21.15.113:0/1154481452 conn(0x55c29e55e410 msgr2=0x55c29e676830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.516+0000 7f078ad51b80 1 -- 172.21.15.113:0/1154481452 shutdown_connections 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.516+0000 7f078ad51b80 1 -- 172.21.15.113:0/1154481452 wait complete. 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.517+0000 7f078ad51b80 1 -- start start 2024-01-15T11:25:31.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.517+0000 7f078ad51b80 1 Processor -- start 2024-01-15T11:25:31.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f078ad51b80 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e6513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f078ad51b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e65ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f078ad51b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 0x55c29e65bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f078ad51b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e583980 con 0x55c29e659410 2024-01-15T11:25:31.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f078ad51b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e523000 con 0x55c29e62edd0 2024-01-15T11:25:31.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f078ad51b80 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c29e522ca0 con 0x55c29e5bcce0 2024-01-15T11:25:31.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f078ad51b80 10 client.0 _ll_register_callbacks cb 0x55c29e386350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f077d0f2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 0x55c29e65bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f077d0f2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 0x55c29e65bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38126/0 (socket says 172.21.15.113:38126) 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f077d0f2700 1 -- 172.21.15.113:0/1564706370 learned_addr learned my addr 172.21.15.113:0/1564706370 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f077c8f1700 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e6513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f0777fff700 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e65ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f077d0f2700 1 -- 172.21.15.113:0/1564706370 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 msgr2=0x55c29e6513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f077d0f2700 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e6513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f077d0f2700 1 -- 172.21.15.113:0/1564706370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 msgr2=0x55c29e65ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f077d0f2700 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e65ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f077d0f2700 1 -- 172.21.15.113:0/1564706370 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0764009640 con 0x55c29e659410 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f077d0f2700 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 0x55c29e65bfc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0770020030 tx=0x7f0770058980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2742e621951a55c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f07577fe700 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f07577fe700 1 -- 172.21.15.113:0/1564706370 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077005c350 con 0x55c29e659410 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f07577fe700 1 -- 172.21.15.113:0/1564706370 <== mon.0 v2:172.21.15.113:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f077006d020 con 0x55c29e659410 2024-01-15T11:25:31.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f078ad51b80 1 -- 172.21.15.113:0/1564706370 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c29e5a3450 con 0x55c29e659410 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f07577fe700 1 -- 172.21.15.113:0/1564706370 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0770058440 con 0x55c29e659410 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f078ad51b80 10 client.4623 fetch_fsmap learned FSMap version 8 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.518+0000 7f078ad51b80 1 -- 172.21.15.113:0/1564706370 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55c29e545290 con 0x55c29e659410 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.519+0000 7f07577fe700 1 -- 172.21.15.113:0/1564706370 <== mon.0 v2:172.21.15.113:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0770073080 con 0x55c29e659410 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.519+0000 7f07577fe700 1 -- 172.21.15.113:0/1564706370 <== mon.0 v2:172.21.15.113:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f07700602b0 con 0x55c29e659410 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.519+0000 7f078ad51b80 10 client.4623 fetch_fsmap finished waiting for FSMap version 8 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.519+0000 7f078ad51b80 10 client.4623 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.519+0000 7f078ad51b80 20 client.4623 populate_metadata read hostname 'smithi113' 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.519+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.141:6833/1012738257,v1:172.21.15.141:6835/1012738257] conn(0x55c29e678500 0x55c29e65e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.519+0000 7f078ad51b80 4 client.4623 mds_command: new command op to 4300 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.519+0000 7f078ad51b80 1 -- 172.21.15.113:0/1564706370 --> [v2:172.21.15.141:6833/1012738257,v1:172.21.15.141:6835/1012738257] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c29e57c960 con 0x55c29e678500 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.519+0000 7f077c8f1700 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.141:6833/1012738257,v1:172.21.15.141:6835/1012738257] conn(0x55c29e678500 0x55c29e65e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.519+0000 7f07577fe700 10 client.4623 ms_handle_connect on v2:172.21.15.141:6833/1012738257 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.519+0000 7f077c8f1700 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.141:6833/1012738257,v1:172.21.15.141:6835/1012738257] conn(0x55c29e678500 0x55c29e65e4d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=133de9eebe9b417a server_cookie=6553db7b338f12bc in_seq=0 out_seq=0 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f07577fe700 1 -- 172.21.15.113:0/1564706370 <== mds.0 v2:172.21.15.141:6833/1012738257 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c29e57c960 con 0x55c29e678500 2024-01-15T11:25:31.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f07577fe700 10 client.4623 handle_command_reply: tid=0 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f078ad51b80 1 client.4623 shutdown 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f078ad51b80 1 -- 172.21.15.113:0/1564706370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 msgr2=0x55c29e65bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 0x55c29e65bfc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0770020030 tx=0x7f0770058980 comp rx=0 tx=0).stop 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f078ad51b80 1 -- 172.21.15.113:0/1564706370 shutdown_connections 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.141:6833/1012738257,v1:172.21.15.141:6835/1012738257] conn(0x55c29e678500 0x55c29e65e4d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e659410 0x55c29e65bfc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e6513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f07577fe700 0 client.4623 ms_handle_reset on v2:172.21.15.141:6833/1012738257 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1564706370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e65ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f078ad51b80 1 -- 172.21.15.113:0/1564706370 >> 172.21.15.113:0/1564706370 conn(0x55c29e55e410 msgr2=0x55c29e62f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f078ad51b80 1 -- 172.21.15.113:0/1564706370 shutdown_connections 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.520+0000 7f078ad51b80 1 -- 172.21.15.113:0/1564706370 wait complete. 2024-01-15T11:25:31.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.521+0000 7f078ad51b80 20 client.4623 trim_cache size 0 max 16384 2024-01-15T11:25:31.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.538+0000 7f078ad51b80 1 Processor -- start 2024-01-15T11:25:31.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.538+0000 7f078ad51b80 1 -- start start 2024-01-15T11:25:31.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f078ad51b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e5bcce0 0x55c29e66f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f078ad51b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e62f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f078ad51b80 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e659410 0x55c29e62f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f078ad51b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e402650 con 0x55c29e5bcce0 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f078ad51b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e62fdd0 con 0x55c29e62edd0 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f078ad51b80 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c29e62f6f0 con 0x55c29e659410 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f077c8f1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e5bcce0 0x55c29e66f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f077c8f1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e5bcce0 0x55c29e66f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38136/0 (socket says 172.21.15.113:38136) 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f077c8f1700 1 -- 172.21.15.113:0/2718706488 learned_addr learned my addr 172.21.15.113:0/2718706488 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f077d0f2700 1 --2- 172.21.15.113:0/2718706488 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e659410 0x55c29e62f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f0777fff700 1 --2- 172.21.15.113:0/2718706488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e62f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f077c8f1700 1 -- 172.21.15.113:0/2718706488 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e659410 msgr2=0x55c29e62f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f077c8f1700 1 --2- 172.21.15.113:0/2718706488 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e659410 0x55c29e62f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f077c8f1700 1 -- 172.21.15.113:0/2718706488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 msgr2=0x55c29e62f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f077c8f1700 1 --2- 172.21.15.113:0/2718706488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e62f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f077c8f1700 1 -- 172.21.15.113:0/2718706488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c29e404c80 con 0x55c29e5bcce0 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f0777fff700 1 --2- 172.21.15.113:0/2718706488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e62f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.539+0000 7f077c8f1700 1 --2- 172.21.15.113:0/2718706488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e5bcce0 0x55c29e66f320 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f076c006f30 tx=0x7f076c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f50e92c74b79b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:31.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.540+0000 7f0762ffd700 1 -- 172.21.15.113:0/2718706488 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c029040 con 0x55c29e5bcce0 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.540+0000 7f0762ffd700 1 -- 172.21.15.113:0/2718706488 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c02a040 con 0x55c29e5bcce0 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.540+0000 7f0762ffd700 1 -- 172.21.15.113:0/2718706488 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c011870 con 0x55c29e5bcce0 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.540+0000 7f078ad51b80 1 -- 172.21.15.113:0/2718706488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e5bcce0 msgr2=0x55c29e66f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.540+0000 7f078ad51b80 1 --2- 172.21.15.113:0/2718706488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e5bcce0 0x55c29e66f320 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f076c006f30 tx=0x7f076c010000 comp rx=0 tx=0).stop 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.540+0000 7f078ad51b80 1 -- 172.21.15.113:0/2718706488 shutdown_connections 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.540+0000 7f078ad51b80 1 --2- 172.21.15.113:0/2718706488 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e659410 0x55c29e62f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.540+0000 7f078ad51b80 1 --2- 172.21.15.113:0/2718706488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e5bcce0 0x55c29e66f320 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.540+0000 7f078ad51b80 1 --2- 172.21.15.113:0/2718706488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e62edd0 0x55c29e62f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.540+0000 7f078ad51b80 1 -- 172.21.15.113:0/2718706488 >> 172.21.15.113:0/2718706488 conn(0x55c29e55e410 msgr2=0x55c29e658830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.540+0000 7f078ad51b80 1 -- 172.21.15.113:0/2718706488 shutdown_connections 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.540+0000 7f078ad51b80 1 -- 172.21.15.113:0/2718706488 wait complete. 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.541+0000 7f078ad51b80 1 -- start start 2024-01-15T11:25:31.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.541+0000 7f078ad51b80 1 Processor -- start 2024-01-15T11:25:31.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f078ad51b80 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e67a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f078ad51b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e62edd0 0x55c29e632800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f078ad51b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e659410 0x55c29e6d61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f078ad51b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e65c2f0 con 0x55c29e62edd0 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f078ad51b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c29e67a890 con 0x55c29e659410 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f078ad51b80 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c29e6e0310 con 0x55c29e5bcce0 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f078ad51b80 10 client.0 _ll_register_callbacks cb 0x55c29e386350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f0777fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e62edd0 0x55c29e632800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f0777fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e62edd0 0x55c29e632800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38138/0 (socket says 172.21.15.113:38138) 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f0777fff700 1 -- 172.21.15.113:0/1185735836 learned_addr learned my addr 172.21.15.113:0/1185735836 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f077c8f1700 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e67a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f077d0f2700 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e659410 0x55c29e6d61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f0777fff700 1 -- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 msgr2=0x55c29e67a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f0777fff700 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e67a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f0777fff700 1 -- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e659410 msgr2=0x55c29e6d61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f0777fff700 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e659410 0x55c29e6d61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f0777fff700 1 -- 172.21.15.113:0/1185735836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0778160850 con 0x55c29e62edd0 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f077c8f1700 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e67a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f0777fff700 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e62edd0 0x55c29e632800 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55c29e677900 tx=0x7f0764011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6785dea2eda1768d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f07577fe700 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f07577fe700 1 -- 172.21.15.113:0/1185735836 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764003880 con 0x55c29e62edd0 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f07577fe700 1 -- 172.21.15.113:0/1185735836 <== mon.0 v2:172.21.15.113:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0764006d40 con 0x55c29e62edd0 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f078ad51b80 1 -- 172.21.15.113:0/1185735836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0764004020 con 0x55c29e62edd0 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f07577fe700 1 -- 172.21.15.113:0/1185735836 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07640039e0 con 0x55c29e62edd0 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f078ad51b80 10 client.4629 fetch_fsmap learned FSMap version 8 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.542+0000 7f078ad51b80 1 -- 172.21.15.113:0/1185735836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55c29e3aafe0 con 0x55c29e62edd0 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f07577fe700 1 -- 172.21.15.113:0/1185735836 <== mon.0 v2:172.21.15.113:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0764004020 con 0x55c29e62edd0 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f07577fe700 1 -- 172.21.15.113:0/1185735836 <== mon.0 v2:172.21.15.113:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0764011070 con 0x55c29e62edd0 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f078ad51b80 10 client.4629 fetch_fsmap finished waiting for FSMap version 8 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f078ad51b80 10 client.4629 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f078ad51b80 20 client.4629 populate_metadata read hostname 'smithi113' 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:6833/1012738257,v1:172.21.15.141:6835/1012738257] conn(0x55c29e68f530 0x55c29e68f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:31.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f078ad51b80 4 client.4629 mds_command: new command op to 4300 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f078ad51b80 1 -- 172.21.15.113:0/1185735836 --> [v2:172.21.15.141:6833/1012738257,v1:172.21.15.141:6835/1012738257] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c29e57c680 con 0x55c29e68f530 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f077c8f1700 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:6833/1012738257,v1:172.21.15.141:6835/1012738257] conn(0x55c29e68f530 0x55c29e68f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f07577fe700 10 client.4629 ms_handle_connect on v2:172.21.15.141:6833/1012738257 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f077c8f1700 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:6833/1012738257,v1:172.21.15.141:6835/1012738257] conn(0x55c29e68f530 0x55c29e68f910 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=291f33555a4a030c server_cookie=3ed42ab6bb30c1a5 in_seq=0 out_seq=0 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f07577fe700 1 -- 172.21.15.113:0/1185735836 <== mds.0 v2:172.21.15.141:6833/1012738257 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55c29e57c680 con 0x55c29e68f530 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f07577fe700 10 client.4629 handle_command_reply: tid=0 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.543+0000 7f078ad51b80 1 client.4629 shutdown 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f078ad51b80 1 -- 172.21.15.113:0/1185735836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e62edd0 msgr2=0x55c29e632800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e62edd0 0x55c29e632800 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55c29e677900 tx=0x7f0764011000 comp rx=0 tx=0).stop 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f078ad51b80 1 -- 172.21.15.113:0/1185735836 shutdown_connections 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c29e659410 0x55c29e6d61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x55c29e5bcce0 0x55c29e67a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.141:6833/1012738257,v1:172.21.15.141:6835/1012738257] conn(0x55c29e68f530 0x55c29e68f910 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f078ad51b80 1 --2- 172.21.15.113:0/1185735836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c29e62edd0 0x55c29e632800 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f07577fe700 0 client.4629 ms_handle_reset on v2:172.21.15.141:6833/1012738257 2024-01-15T11:25:31.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f078ad51b80 1 -- 172.21.15.113:0/1185735836 >> 172.21.15.113:0/1185735836 conn(0x55c29e55e410 msgr2=0x55c29e62b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:31.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f078ad51b80 1 -- 172.21.15.113:0/1185735836 shutdown_connections 2024-01-15T11:25:31.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f078ad51b80 1 -- 172.21.15.113:0/1185735836 wait complete. 2024-01-15T11:25:31.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f078ad51b80 20 client.4629 trim_cache size 0 max 16384 2024-01-15T11:25:31.548 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:25:31.548 INFO:teuthology.orchestra.run.smithi113.stdout: "cluster_fsid": "910f8628-cd75-4a37-bf4e-e3603a95290d", 2024-01-15T11:25:31.548 INFO:teuthology.orchestra.run.smithi113.stdout: "whoami": 0, 2024-01-15T11:25:31.548 INFO:teuthology.orchestra.run.smithi113.stdout: "id": 4300, 2024-01-15T11:25:31.548 INFO:teuthology.orchestra.run.smithi113.stdout: "want_state": "up:active", 2024-01-15T11:25:31.548 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "up:active", 2024-01-15T11:25:31.548 INFO:teuthology.orchestra.run.smithi113.stdout: "fs_name": "cephfs", 2024-01-15T11:25:31.548 INFO:teuthology.orchestra.run.smithi113.stdout: "rank_uptime": 15.294442135000001, 2024-01-15T11:25:31.549 INFO:teuthology.orchestra.run.smithi113.stdout: "mdsmap_epoch": 8, 2024-01-15T11:25:31.549 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_epoch": 20, 2024-01-15T11:25:31.549 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_epoch_barrier": 20, 2024-01-15T11:25:31.549 INFO:teuthology.orchestra.run.smithi113.stdout: "uptime": 22.877507137999999 2024-01-15T11:25:31.549 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:25:31.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f077eb55700 1 -- 172.21.15.113:0/4044344059 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f075c06b350 msgr2=0x7f075c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:31.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f077eb55700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f075c06b350 0x7f075c06d810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f076c0090b0 tx=0x7f076c01c040 comp rx=0 tx=0).stop 2024-01-15T11:25:31.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f077eb55700 1 -- 172.21.15.113:0/4044344059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781603c0 msgr2=0x7f07780b56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:31.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f077eb55700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781603c0 0x7f07780b56c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f077000cca0 tx=0x7f077000a060 comp rx=0 tx=0).stop 2024-01-15T11:25:31.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f077eb55700 1 -- 172.21.15.113:0/4044344059 shutdown_connections 2024-01-15T11:25:31.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f077eb55700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f075c06b350 0x7f075c06d810 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f077eb55700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07781603c0 0x7f07780b56c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f077eb55700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f07780d2160 0x7f07780c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f077eb55700 1 --2- 172.21.15.113:0/4044344059 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f077815bed0 0x7f07780b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:31.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f077eb55700 1 -- 172.21.15.113:0/4044344059 >> 172.21.15.113:0/4044344059 conn(0x7f0778073130 msgr2=0x7f07780b3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:31.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f077eb55700 1 -- 172.21.15.113:0/4044344059 shutdown_connections 2024-01-15T11:25:31.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:31.544+0000 7f077eb55700 1 -- 172.21.15.113:0/4044344059 wait complete. 2024-01-15T11:25:31.555 INFO:teuthology.run_tasks:Running task kclient... 2024-01-15T11:25:31.602 INFO:tasks.kclient:Mounting kernel clients... 2024-01-15T11:25:31.602 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-01-15T11:25:31.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:31.603 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-01-15T11:25:31.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:31.620 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link delete ceph-brx 2024-01-15T11:25:31.689 INFO:teuthology.orchestra.run.smithi113.stderr:Cannot find device "ceph-brx" 2024-01-15T11:25:31.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T11:25:31.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:31.691 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-01-15T11:25:31.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:31.749 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link delete ceph-brx 2024-01-15T11:25:31.817 INFO:teuthology.orchestra.run.smithi113.stderr:Cannot find device "ceph-brx" 2024-01-15T11:25:31.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T11:25:31.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:31.819 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-15T11:25:31.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:31.842 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link delete ceph-brx 2024-01-15T11:25:31.917 INFO:teuthology.orchestra.run.smithi141.stderr:Cannot find device "ceph-brx" 2024-01-15T11:25:31.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T11:25:31.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:31.919 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-15T11:25:31.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:31.979 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link delete ceph-brx 2024-01-15T11:25:32.052 INFO:teuthology.orchestra.run.smithi141.stderr:Cannot find device "ceph-brx" 2024-01-15T11:25:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T11:25:32.053 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-01-15T11:25:32.054 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T11:25:32.054 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T11:25:32.054 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T11:25:32.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.178+0000 7f2880036700 1 Processor -- start 2024-01-15T11:25:32.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.178+0000 7f2880036700 1 -- start start 2024-01-15T11:25:32.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f2880036700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f28780b19f0 0x7f28781603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:32.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f2880036700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28780cd100 con 0x7f287815bed0 2024-01-15T11:25:32.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f2880036700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28780d09a0 con 0x7f28780d2160 2024-01-15T11:25:32.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f2880036700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28780b1e80 con 0x7f28780b19f0 2024-01-15T11:25:32.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287ddd2700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f287815bed0 0x7f28780d1a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56678/0 (socket says 172.21.15.113:56678) 2024-01-15T11:25:32.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287ddd2700 1 -- 172.21.15.113:0/2045196837 learned_addr learned my addr 172.21.15.113:0/2045196837 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:32.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287e5d3700 1 --2- 172.21.15.113:0/2045196837 >> v2:172.21.15.141:3300/0 conn(0x7f28780b19f0 0x7f28781603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909665247 0 0) 0x7f28780cd100 con 0x7f287815bed0 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28640035b0 con 0x7f287815bed0 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3270933397 0 0) 0x7f28640035b0 con 0x7f287815bed0 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 >> v2:172.21.15.141:3300/0 conn(0x7f28780b19f0 msgr2=0x7f28781603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 --2- 172.21.15.113:0/2045196837 >> v2:172.21.15.141:3300/0 conn(0x7f28780b19f0 0x7f28781603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 >> v1:172.21.15.141:6789/0 conn(0x7f28780d2160 legacy=0x7f28780cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28780cc7d0 con 0x7f287815bed0 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f2874001670 con 0x7f287815bed0 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 >> v1:172.21.15.113:6789/0 conn(0x7f287815bed0 legacy=0x7f28780d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 --2- 172.21.15.113:0/2045196837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2864003a00 0x7f2864003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 --2- 172.21.15.113:0/2045196837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 0x7f2864004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 --2- 172.21.15.113:0/2045196837 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2864005210 0x7f286400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28780b1e80 con 0x7f2864004850 2024-01-15T11:25:32.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286400c0b0 con 0x7f2864003a00 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286400c270 con 0x7f2864005210 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.179+0000 7f287d5d1700 1 --2- 172.21.15.113:0/2045196837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 0x7f2864004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.180+0000 7f287ddd2700 1 --2- 172.21.15.113:0/2045196837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2864003a00 0x7f2864003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.180+0000 7f287e5d3700 1 --2- 172.21.15.113:0/2045196837 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2864005210 0x7f286400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.180+0000 7f287d5d1700 1 -- 172.21.15.113:0/2045196837 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2864005210 msgr2=0x7f286400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.180+0000 7f287d5d1700 1 --2- 172.21.15.113:0/2045196837 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2864005210 0x7f286400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.180+0000 7f287d5d1700 1 -- 172.21.15.113:0/2045196837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2864003a00 msgr2=0x7f2864003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.180+0000 7f287d5d1700 1 --2- 172.21.15.113:0/2045196837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2864003a00 0x7f2864003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.180+0000 7f287e5d3700 1 -- 172.21.15.113:0/2045196837 reap_dead start 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.180+0000 7f287d5d1700 1 -- 172.21.15.113:0/2045196837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28640035b0 con 0x7f2864004850 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.180+0000 7f287d5d1700 1 --2- 172.21.15.113:0/2045196837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 0x7f2864004cd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f28780d25f0 tx=0x7f2864005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e91f4269e20931e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.180+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28680052d0 con 0x7f2864004850 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.180+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2868005450 con 0x7f2864004850 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.180+0000 7f287cdd0700 1 -- 172.21.15.113:0/2045196837 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28680038c0 con 0x7f2864004850 2024-01-15T11:25:32.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.181+0000 7f2880036700 1 -- 172.21.15.113:0/2045196837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 msgr2=0x7f2864004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:32.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.181+0000 7f2880036700 1 --2- 172.21.15.113:0/2045196837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 0x7f2864004cd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f28780d25f0 tx=0x7f2864005880 comp rx=0 tx=0).stop 2024-01-15T11:25:32.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.181+0000 7f2880036700 1 -- 172.21.15.113:0/2045196837 shutdown_connections 2024-01-15T11:25:32.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.181+0000 7f2880036700 1 --2- 172.21.15.113:0/2045196837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 0x7f2864004cd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:32.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.181+0000 7f2880036700 1 -- 172.21.15.113:0/2045196837 >> 172.21.15.113:0/2045196837 conn(0x7f2878073130 msgr2=0x7f287809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:32.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.181+0000 7f2880036700 1 -- 172.21.15.113:0/2045196837 shutdown_connections 2024-01-15T11:25:32.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.181+0000 7f2880036700 1 -- 172.21.15.113:0/2045196837 wait complete. 2024-01-15T11:25:32.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.182+0000 7f2880036700 1 Processor -- start 2024-01-15T11:25:32.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.182+0000 7f2880036700 1 -- start start 2024-01-15T11:25:32.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f2880036700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 0x7f287815b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f2880036700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f28780c5780 0x7f28780b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f2880036700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28780b5e00 0x7f28780cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f2880036700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28780b69f0 con 0x7f2864004850 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f2880036700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28780c7550 con 0x7f28780b5e00 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f2880036700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28780b7340 con 0x7f28780c5780 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f287ddd2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 0x7f287815b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f287ddd2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 0x7f287815b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38152/0 (socket says 172.21.15.113:38152) 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f287ddd2700 1 -- 172.21.15.113:0/4192225128 learned_addr learned my addr 172.21.15.113:0/4192225128 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f287d5d1700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f28780c5780 0x7f28780b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f287e5d3700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28780b5e00 0x7f28780cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f287ddd2700 1 -- 172.21.15.113:0/4192225128 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f28780c5780 msgr2=0x7f28780b58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f287ddd2700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f28780c5780 0x7f28780b58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f287ddd2700 1 -- 172.21.15.113:0/4192225128 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28780b5e00 msgr2=0x7f28780cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f287ddd2700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28780b5e00 0x7f28780cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f287ddd2700 1 -- 172.21.15.113:0/4192225128 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28780b6430 con 0x7f2864004850 2024-01-15T11:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f287ddd2700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 0x7f287815b180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f28740072c0 tx=0x7f28740075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f286e7fc700 1 -- 172.21.15.113:0/4192225128 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874011580 con 0x7f2864004850 2024-01-15T11:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f286e7fc700 1 -- 172.21.15.113:0/4192225128 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28740116e0 con 0x7f2864004850 2024-01-15T11:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f2880036700 1 -- 172.21.15.113:0/4192225128 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28780b6710 con 0x7f2864004850 2024-01-15T11:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f286e7fc700 1 -- 172.21.15.113:0/4192225128 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874011a90 con 0x7f2864004850 2024-01-15T11:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.183+0000 7f2880036700 1 -- 172.21.15.113:0/4192225128 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287815b6c0 con 0x7f2864004850 2024-01-15T11:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.184+0000 7f2880036700 1 -- 172.21.15.113:0/4192225128 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28780b61e0 con 0x7f2864004850 2024-01-15T11:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.185+0000 7f286e7fc700 1 -- 172.21.15.113:0/4192225128 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2874011bf0 con 0x7f2864004850 2024-01-15T11:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.185+0000 7f286e7fc700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f285c06b350 0x7f285c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.185+0000 7f286e7fc700 1 -- 172.21.15.113:0/4192225128 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2874089410 con 0x7f2864004850 2024-01-15T11:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.185+0000 7f287d5d1700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f285c06b350 0x7f285c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.185+0000 7f287d5d1700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f285c06b350 0x7f285c06d810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2868000c00 tx=0x7f2868015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:32.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.189+0000 7f286e7fc700 1 -- 172.21.15.113:0/4192225128 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2874054310 con 0x7f2864004850 2024-01-15T11:25:32.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.428+0000 7f2880036700 1 -- 172.21.15.113:0/4192225128 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2878067f60 con 0x7f2864004850 2024-01-15T11:25:32.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.429+0000 7f286e7fc700 1 -- 172.21.15.113:0/4192225128 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f287405ce60 con 0x7f2864004850 2024-01-15T11:25:32.428 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T11:25:32.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.431+0000 7f2880036700 1 -- 172.21.15.113:0/4192225128 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f285c06b350 msgr2=0x7f285c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:32.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.431+0000 7f2880036700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f285c06b350 0x7f285c06d810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2868000c00 tx=0x7f2868015000 comp rx=0 tx=0).stop 2024-01-15T11:25:32.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.431+0000 7f2880036700 1 -- 172.21.15.113:0/4192225128 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 msgr2=0x7f287815b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:32.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.431+0000 7f2880036700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 0x7f287815b180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f28740072c0 tx=0x7f28740075d0 comp rx=0 tx=0).stop 2024-01-15T11:25:32.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.431+0000 7f2880036700 1 -- 172.21.15.113:0/4192225128 shutdown_connections 2024-01-15T11:25:32.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.431+0000 7f2880036700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f285c06b350 0x7f285c06d810 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:32.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.431+0000 7f2880036700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28780b5e00 0x7f28780cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:32.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.431+0000 7f2880036700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2864004850 0x7f287815b180 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:32.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.431+0000 7f2880036700 1 --2- 172.21.15.113:0/4192225128 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f28780c5780 0x7f28780b58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:32.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.431+0000 7f2880036700 1 -- 172.21.15.113:0/4192225128 >> 172.21.15.113:0/4192225128 conn(0x7f2878073130 msgr2=0x7f28780cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:32.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.431+0000 7f2880036700 1 -- 172.21.15.113:0/4192225128 shutdown_connections 2024-01-15T11:25:32.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:32.431+0000 7f2880036700 1 -- 172.21.15.113:0/4192225128 wait complete. 2024-01-15T11:25:32.441 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T11:25:32.442 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T11:25:32.442 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi113.front.sepia.ceph.com 2024-01-15T11:25:32.442 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T11:25:32.442 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T11:25:32.442 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T11:25:32.442 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T11:25:32.442 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T11:25:32.442 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T11:25:32.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:32.442 DEBUG:teuthology.orchestra.run.smithi113:> ip addr 2024-01-15T11:25:32.459 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T11:25:32.459 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T11:25:32.459 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T11:25:32.460 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:32.460 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-01-15T11:25:32.460 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:32.460 INFO:teuthology.orchestra.run.smithi113.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:25:32.460 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:44 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:32.460 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f0 2024-01-15T11:25:32.460 INFO:teuthology.orchestra.run.smithi113.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:25:32.460 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:45 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:32.460 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f1 2024-01-15T11:25:32.460 INFO:teuthology.orchestra.run.smithi113.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:25:32.460 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c0 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:32.460 INFO:teuthology.orchestra.run.smithi113.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T11:25:32.461 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c1 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:32.461 INFO:teuthology.orchestra.run.smithi113.stdout: inet 172.21.15.113/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T11:25:32.461 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft 42492sec preferred_lft 42492sec 2024-01-15T11:25:32.461 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::105b:ecb2:15ad:83c/64 scope link noprefixroute 2024-01-15T11:25:32.461 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:32.461 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-15T11:25:32.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:32.461 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-15T11:25:32.461 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-15T11:25:32.462 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link add name ceph-brx type bridge 2024-01-15T11:25:32.462 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip addr flush dev ceph-brx 2024-01-15T11:25:32.462 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set ceph-brx up 2024-01-15T11:25:32.462 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-15T11:25:32.462 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-15T11:25:32.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:32.637 DEBUG:teuthology.orchestra.run.smithi113:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T11:25:32.664 INFO:teuthology.orchestra.run.smithi113.stdout:1 2024-01-15T11:25:32.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:32.666 DEBUG:teuthology.orchestra.run.smithi113:> route 2024-01-15T11:25:32.742 INFO:teuthology.orchestra.run.smithi113.stdout:Kernel IP routing table 2024-01-15T11:25:32.742 INFO:teuthology.orchestra.run.smithi113.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T11:25:32.742 INFO:teuthology.orchestra.run.smithi113.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T11:25:32.742 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T11:25:32.742 INFO:teuthology.orchestra.run.smithi113.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T11:25:32.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:32.743 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-15T11:25:32.743 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-15T11:25:32.743 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T11:25:32.743 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T11:25:32.743 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T11:25:32.743 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-15T11:25:32.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:32.973 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-01-15T11:25:32.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:32.990 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list-id 2024-01-15T11:25:33.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:33.048 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-15T11:25:33.048 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-15T11:25:33.048 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T11:25:33.048 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-15T11:25:33.048 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-15T11:25:33.142 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-15T11:25:33.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:33.143 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-15T11:25:33.143 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-15T11:25:33.143 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-15T11:25:33.143 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-15T11:25:33.143 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-15T11:25:33.143 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-15T11:25:33.143 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-15T11:25:33.143 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-15T11:25:33.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:33.362 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-15T11:25:33.362 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-15T11:25:33.362 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set brx.0 up 2024-01-15T11:25:33.362 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set dev brx.0 master ceph-brx 2024-01-15T11:25:33.363 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-15T11:25:33.460 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T11:25:33.461 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T11:25:33.478 INFO:teuthology.orchestra.run.smithi113.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T11:25:33.478 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T11:25:33.478 DEBUG:teuthology.orchestra.run.smithi113:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T11:25:33.535 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-15T11:25:33.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:33.535 DEBUG:teuthology.orchestra.run.smithi113:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-15T11:25:33.621 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-15T11:25:33.622 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-15T11:25:33.622 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-15T11:25:33.720 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-15T11:25:33.720 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-15T11:25:33.720 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-15T11:25:33.720 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.141:3300,172.21.15.141:3301" 2024-01-15T11:25:33.720 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T11:25:33.739 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-15T11:25:33.740 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-15T11:25:33.740 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-15T11:25:33.740 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.141:3300,172.21.15.141:3301" 2024-01-15T11:25:33.740 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T11:25:33.740 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:6789,172.21.15.141:6789,172.21.15.141:6790" 2024-01-15T11:25:33.740 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.113:6789,172.21.15.141:6789,172.21.15.141:6790:/ 2024-01-15T11:25:33.740 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=910f8628-cd75-4a37-bf4e-e3603a95290d" will pass to kernel 2024-01-15T11:25:33.742 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-15T11:25:33.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:33.743 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T11:25:33.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:33.771 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:25:33.858 INFO:teuthology.orchestra.run.smithi113.stdout:client4466 2024-01-15T11:25:33.859 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:25:33.859 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:33.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:33.879 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:25:33.972 INFO:teuthology.orchestra.run.smithi113.stdout:client4466 2024-01-15T11:25:33.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:33.973 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:25:34.028 INFO:teuthology.orchestra.run.smithi113.stdout:client4466 2024-01-15T11:25:34.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:34.028 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/910f8628-cd75-4a37-bf4e-e3603a95290d.client4466/status') 2024-01-15T11:25:34.089 INFO:teuthology.orchestra.run.smithi113.stdout:instance: client.4466 (3)172.21.15.113:0/1310496093 2024-01-15T11:25:34.089 INFO:teuthology.orchestra.run.smithi113.stdout:blocklisted: false 2024-01-15T11:25:34.089 INFO:teuthology.orchestra.run.smithi113.stderr:0+1 records in 2024-01-15T11:25:34.089 INFO:teuthology.orchestra.run.smithi113.stderr:0+1 records out 2024-01-15T11:25:34.089 INFO:teuthology.orchestra.run.smithi113.stderr:71 bytes copied, 2.0474e-05 s, 3.5 MB/s 2024-01-15T11:25:34.090 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-01-15T11:25:34.091 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T11:25:34.091 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-15T11:25:34.091 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T11:25:34.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.213+0000 7fc5ccc05700 1 Processor -- start 2024-01-15T11:25:34.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.213+0000 7fc5ccc05700 1 -- start start 2024-01-15T11:25:34.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.214+0000 7fc5ccc05700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc5c80aedb0 0x7fc5c80b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:34.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.214+0000 7fc5ccc05700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c8073520 con 0x7fc5c815aee0 2024-01-15T11:25:34.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.214+0000 7fc5ccc05700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80adbc0 con 0x7fc5c80b3c30 2024-01-15T11:25:34.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.214+0000 7fc5ccc05700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c80ae280 con 0x7fc5c80aedb0 2024-01-15T11:25:34.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.214+0000 7fc5c6d9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc5c80aedb0 0x7fc5c80b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:34.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.214+0000 7fc5c659c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fc5c815aee0 0x7fc5c80af9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56690/0 (socket says 172.21.15.113:56690) 2024-01-15T11:25:34.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.214+0000 7fc5c659c700 1 -- 172.21.15.113:0/1723117258 learned_addr learned my addr 172.21.15.113:0/1723117258 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:34.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.214+0000 7fc5c6d9d700 1 -- 172.21.15.113:0/1723117258 >> v1:172.21.15.141:6789/0 conn(0x7fc5c80b3c30 legacy=0x7fc5c80ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:34.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.214+0000 7fc5c6d9d700 1 -- 172.21.15.113:0/1723117258 >> v1:172.21.15.113:6789/0 conn(0x7fc5c815aee0 legacy=0x7fc5c80af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:34.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.214+0000 7fc5c6d9d700 1 -- 172.21.15.113:0/1723117258 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80adea0 con 0x7fc5c80aedb0 2024-01-15T11:25:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.214+0000 7fc5c559a700 1 -- 172.21.15.113:0/1723117258 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035087065 0 0) 0x7fc5c8073520 con 0x7fc5c815aee0 2024-01-15T11:25:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.215+0000 7fc5c6d9d700 1 --2- 172.21.15.113:0/1723117258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5c80aedb0 0x7fc5c80b40e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009d90 tx=0x7fc5c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a4990df3e7d2cd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.215+0000 7fc5c559a700 1 -- 172.21.15.113:0/1723117258 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c000e070 con 0x7fc5c80aedb0 2024-01-15T11:25:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.215+0000 7fc5c559a700 1 -- 172.21.15.113:0/1723117258 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c000bb90 con 0x7fc5c80aedb0 2024-01-15T11:25:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.215+0000 7fc5c559a700 1 -- 172.21.15.113:0/1723117258 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0003a20 con 0x7fc5c80aedb0 2024-01-15T11:25:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.215+0000 7fc5ccc05700 1 -- 172.21.15.113:0/1723117258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5c80aedb0 msgr2=0x7fc5c80b40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.215+0000 7fc5ccc05700 1 --2- 172.21.15.113:0/1723117258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5c80aedb0 0x7fc5c80b40e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009d90 tx=0x7fc5c000b330 comp rx=0 tx=0).stop 2024-01-15T11:25:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.215+0000 7fc5ccc05700 1 -- 172.21.15.113:0/1723117258 shutdown_connections 2024-01-15T11:25:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.215+0000 7fc5ccc05700 1 --2- 172.21.15.113:0/1723117258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5c80aedb0 0x7fc5c80b40e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.215+0000 7fc5ccc05700 1 -- 172.21.15.113:0/1723117258 >> 172.21.15.113:0/1723117258 conn(0x7fc5c80b5640 msgr2=0x7fc5c80b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.215+0000 7fc5ccc05700 1 -- 172.21.15.113:0/1723117258 shutdown_connections 2024-01-15T11:25:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.215+0000 7fc5ccc05700 1 -- 172.21.15.113:0/1723117258 wait complete. 2024-01-15T11:25:34.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.217+0000 7fc5ccc05700 1 Processor -- start 2024-01-15T11:25:34.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.217+0000 7fc5ccc05700 1 -- start start 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.217+0000 7fc5ccc05700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5c80aedb0 0x7fc5c815d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.217+0000 7fc5ccc05700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5c80b3c30 0x7fc5c815d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.217+0000 7fc5ccc05700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5c815aee0 0x7fc5c815bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.217+0000 7fc5ccc05700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c80b4620 con 0x7fc5c80aedb0 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.217+0000 7fc5ccc05700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c81522d0 con 0x7fc5c815aee0 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.217+0000 7fc5ccc05700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c814f220 con 0x7fc5c80b3c30 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5c659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5c80aedb0 0x7fc5c815d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5c659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5c80aedb0 0x7fc5c815d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38176/0 (socket says 172.21.15.113:38176) 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5c659c700 1 -- 172.21.15.113:0/3005836206 learned_addr learned my addr 172.21.15.113:0/3005836206 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5c6d9d700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5c815aee0 0x7fc5c815bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5c659c700 1 -- 172.21.15.113:0/3005836206 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5c80b3c30 msgr2=0x7fc5c815d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5c5d9b700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5c80b3c30 0x7fc5c815d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:34.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5c659c700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5c80b3c30 0x7fc5c815d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:34.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5c659c700 1 -- 172.21.15.113:0/3005836206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5c815aee0 msgr2=0x7fc5c815bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:34.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5c659c700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5c815aee0 0x7fc5c815bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:34.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5c659c700 1 -- 172.21.15.113:0/3005836206 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c0009a70 con 0x7fc5c80aedb0 2024-01-15T11:25:34.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5c6d9d700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5c815aee0 0x7fc5c815bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:34.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5c659c700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5c80aedb0 0x7fc5c815d490 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc00c520 tx=0x7fc5bc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:34.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5b6ffd700 1 -- 172.21.15.113:0/3005836206 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc004c80 con 0x7fc5c80aedb0 2024-01-15T11:25:34.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5b6ffd700 1 -- 172.21.15.113:0/3005836206 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5bc004e20 con 0x7fc5c80aedb0 2024-01-15T11:25:34.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5ccc05700 1 -- 172.21.15.113:0/3005836206 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c814fda0 con 0x7fc5c80aedb0 2024-01-15T11:25:34.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5b6ffd700 1 -- 172.21.15.113:0/3005836206 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc007310 con 0x7fc5c80aedb0 2024-01-15T11:25:34.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.218+0000 7fc5ccc05700 1 -- 172.21.15.113:0/3005836206 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c814f750 con 0x7fc5c80aedb0 2024-01-15T11:25:34.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.219+0000 7fc5ccc05700 1 -- 172.21.15.113:0/3005836206 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c8067f60 con 0x7fc5c80aedb0 2024-01-15T11:25:34.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.220+0000 7fc5b6ffd700 1 -- 172.21.15.113:0/3005836206 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc5bc007470 con 0x7fc5c80aedb0 2024-01-15T11:25:34.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.220+0000 7fc5b6ffd700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc5a806b350 0x7fc5a806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:34.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.220+0000 7fc5b6ffd700 1 -- 172.21.15.113:0/3005836206 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5bc0897e0 con 0x7fc5c80aedb0 2024-01-15T11:25:34.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.220+0000 7fc5c5d9b700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc5a806b350 0x7fc5a806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:34.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.220+0000 7fc5c5d9b700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc5a806b350 0x7fc5a806d810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc5c806ee80 tx=0x7fc5b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:34.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.224+0000 7fc5b6ffd700 1 -- 172.21.15.113:0/3005836206 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5bc0546e0 con 0x7fc5c80aedb0 2024-01-15T11:25:34.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.462+0000 7fc5ccc05700 1 -- 172.21.15.113:0/3005836206 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc5c8061e70 con 0x7fc5c80aedb0 2024-01-15T11:25:34.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.462+0000 7fc5b6ffd700 1 -- 172.21.15.113:0/3005836206 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc5bc057d00 con 0x7fc5c80aedb0 2024-01-15T11:25:34.461 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T11:25:34.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.464+0000 7fc5ccc05700 1 -- 172.21.15.113:0/3005836206 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc5a806b350 msgr2=0x7fc5a806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:34.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.464+0000 7fc5ccc05700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc5a806b350 0x7fc5a806d810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc5c806ee80 tx=0x7fc5b8009040 comp rx=0 tx=0).stop 2024-01-15T11:25:34.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.464+0000 7fc5ccc05700 1 -- 172.21.15.113:0/3005836206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5c80aedb0 msgr2=0x7fc5c815d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:34.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.464+0000 7fc5ccc05700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5c80aedb0 0x7fc5c815d490 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc00c520 tx=0x7fc5bc00c8e0 comp rx=0 tx=0).stop 2024-01-15T11:25:34.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.464+0000 7fc5ccc05700 1 -- 172.21.15.113:0/3005836206 shutdown_connections 2024-01-15T11:25:34.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.464+0000 7fc5ccc05700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc5a806b350 0x7fc5a806d810 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:34.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.464+0000 7fc5ccc05700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc5c815aee0 0x7fc5c815bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:34.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.464+0000 7fc5ccc05700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5c80aedb0 0x7fc5c815d490 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:34.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.464+0000 7fc5ccc05700 1 --2- 172.21.15.113:0/3005836206 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5c80b3c30 0x7fc5c815d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:34.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.464+0000 7fc5ccc05700 1 -- 172.21.15.113:0/3005836206 >> 172.21.15.113:0/3005836206 conn(0x7fc5c80b5640 msgr2=0x7fc5c80b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:34.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.464+0000 7fc5ccc05700 1 -- 172.21.15.113:0/3005836206 shutdown_connections 2024-01-15T11:25:34.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:34.464+0000 7fc5ccc05700 1 -- 172.21.15.113:0/3005836206 wait complete. 2024-01-15T11:25:34.475 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T11:25:34.475 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T11:25:34.475 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi113.front.sepia.ceph.com 2024-01-15T11:25:34.475 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-01-15T11:25:34.475 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-15T11:25:34.475 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T11:25:34.475 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T11:25:34.475 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T11:25:34.475 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-01-15T11:25:34.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:34.476 DEBUG:teuthology.orchestra.run.smithi113:> ip addr 2024-01-15T11:25:34.493 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T11:25:34.493 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T11:25:34.493 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T11:25:34.493 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:34.493 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-01-15T11:25:34.493 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:34.493 INFO:teuthology.orchestra.run.smithi113.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:25:34.493 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:44 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:34.493 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f0 2024-01-15T11:25:34.493 INFO:teuthology.orchestra.run.smithi113.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:25:34.493 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:45 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f1 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c0 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c1 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout: inet 172.21.15.113/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft 42490sec preferred_lft 42490sec 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::105b:ecb2:15ad:83c/64 scope link noprefixroute 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 26:9b:2b:80:b1:f2 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::b017:17ff:fe98:c0a9/64 scope link 2024-01-15T11:25:34.494 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:34.495 INFO:teuthology.orchestra.run.smithi113.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T11:25:34.495 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 26:9b:2b:80:b1:f2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T11:25:34.495 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::249b:2bff:fe80:b1f2/64 scope link tentative 2024-01-15T11:25:34.495 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:34.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:34.495 DEBUG:teuthology.orchestra.run.smithi113:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T11:25:34.560 INFO:teuthology.orchestra.run.smithi113.stdout:1 2024-01-15T11:25:34.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:34.561 DEBUG:teuthology.orchestra.run.smithi113:> route 2024-01-15T11:25:34.620 INFO:teuthology.orchestra.run.smithi113.stdout:Kernel IP routing table 2024-01-15T11:25:34.620 INFO:teuthology.orchestra.run.smithi113.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T11:25:34.621 INFO:teuthology.orchestra.run.smithi113.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T11:25:34.621 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T11:25:34.621 INFO:teuthology.orchestra.run.smithi113.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T11:25:34.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:34.621 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-15T11:25:34.621 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-15T11:25:34.621 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T11:25:34.621 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T11:25:34.621 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T11:25:34.621 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-15T11:25:34.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:34.725 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-01-15T11:25:34.742 INFO:teuthology.orchestra.run.smithi113.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T11:25:34.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:34.742 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list-id 2024-01-15T11:25:34.799 INFO:teuthology.orchestra.run.smithi113.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-01-15T11:25:34.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:34.800 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-15T11:25:34.800 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-15T11:25:34.800 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T11:25:34.800 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-01-15T11:25:34.800 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-15T11:25:34.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:34.894 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-15T11:25:34.928 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T11:25:34.928 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T11:25:34.928 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T11:25:34.928 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:34.929 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-01-15T11:25:34.929 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:34.929 INFO:teuthology.orchestra.run.smithi113.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T11:25:34.929 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 3e:68:81:a8:a7:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T11:25:34.929 INFO:teuthology.orchestra.run.smithi113.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T11:25:34.929 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:34.929 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::3c68:81ff:fea8:a7f9/64 scope link tentative 2024-01-15T11:25:34.929 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:34.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:34.941 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-15T11:25:35.016 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T11:25:35.017 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T11:25:35.017 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T11:25:35.017 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:35.017 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-01-15T11:25:35.017 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:35.017 INFO:teuthology.orchestra.run.smithi113.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T11:25:35.017 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 3e:68:81:a8:a7:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T11:25:35.017 INFO:teuthology.orchestra.run.smithi113.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T11:25:35.017 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:35.017 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::3c68:81ff:fea8:a7f9/64 scope link tentative 2024-01-15T11:25:35.017 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:35.029 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-01-15T11:25:35.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:35.029 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-15T11:25:35.029 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-15T11:25:35.029 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-01-15T11:25:35.029 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-15T11:25:35.029 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-01-15T11:25:35.029 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-01-15T11:25:35.029 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-01-15T11:25:35.029 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-15T11:25:35.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:35.279 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-15T11:25:35.279 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-15T11:25:35.279 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set brx.1 up 2024-01-15T11:25:35.279 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set dev brx.1 master ceph-brx 2024-01-15T11:25:35.279 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-15T11:25:35.376 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T11:25:35.377 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-01-15T11:25:35.393 INFO:teuthology.orchestra.run.smithi113.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-01-15T11:25:35.394 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T11:25:35.394 DEBUG:teuthology.orchestra.run.smithi113:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-01-15T11:25:35.450 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-15T11:25:35.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:35.450 DEBUG:teuthology.orchestra.run.smithi113:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-01-15T11:25:35.537 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-01-15T11:25:35.537 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-01-15T11:25:35.537 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-15T11:25:35.546 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-01-15T11:25:35.546 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-01-15T11:25:35.546 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-15T11:25:35.546 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.141:3300,172.21.15.141:3301" 2024-01-15T11:25:35.546 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T11:25:35.565 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-01-15T11:25:35.565 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-01-15T11:25:35.565 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-15T11:25:35.565 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.141:3300,172.21.15.141:3301" 2024-01-15T11:25:35.565 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T11:25:35.565 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:6789,172.21.15.141:6789,172.21.15.141:6790" 2024-01-15T11:25:35.565 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.113:6789,172.21.15.141:6789,172.21.15.141:6790:/ 2024-01-15T11:25:35.565 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=910f8628-cd75-4a37-bf4e-e3603a95290d" will pass to kernel 2024-01-15T11:25:35.568 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-15T11:25:35.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:35.568 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-01-15T11:25:35.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:35.595 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:25:35.667 INFO:teuthology.orchestra.run.smithi113.stdout:client4469 2024-01-15T11:25:35.668 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:25:35.668 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:35.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:35.726 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:25:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:client4469 2024-01-15T11:25:35.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:35.799 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:25:35.871 INFO:teuthology.orchestra.run.smithi113.stdout:client4469 2024-01-15T11:25:35.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:35.872 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/910f8628-cd75-4a37-bf4e-e3603a95290d.client4469/status') 2024-01-15T11:25:35.952 INFO:teuthology.orchestra.run.smithi113.stdout:instance: client.4469 (3)172.21.15.113:0/2785813958 2024-01-15T11:25:35.952 INFO:teuthology.orchestra.run.smithi113.stdout:blocklisted: false 2024-01-15T11:25:35.952 INFO:teuthology.orchestra.run.smithi113.stderr:0+1 records in 2024-01-15T11:25:35.952 INFO:teuthology.orchestra.run.smithi113.stderr:0+1 records out 2024-01-15T11:25:35.952 INFO:teuthology.orchestra.run.smithi113.stderr:71 bytes copied, 2.1728e-05 s, 3.3 MB/s 2024-01-15T11:25:35.953 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-01-15T11:25:35.953 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T11:25:35.954 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T11:25:35.954 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T11:25:36.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.116+0000 7f8dfad67700 1 Processor -- start 2024-01-15T11:25:36.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.116+0000 7f8dfad67700 1 -- start start 2024-01-15T11:25:36.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8dfad67700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df40cfb00 0x7f8df40cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:36.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8dfad67700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df40cf060 con 0x7f8df4073f90 2024-01-15T11:25:36.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8dfad67700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df4160330 con 0x7f8df406dc20 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8dfad67700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df41608f0 con 0x7f8df40cfb00 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df8b03700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f8df4073f90 0x7f8df4074370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42486/0 (socket says 172.21.15.113:42486) 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df8b03700 1 -- 172.21.15.113:0/1160681840 learned_addr learned my addr 172.21.15.113:0/1160681840 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df9304700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df40cfb00 0x7f8df40cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131914280 0 0) 0x7f8df40cf060 con 0x7f8df4073f90 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de40035b0 con 0x7f8df4073f90 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 705287615 0 0) 0x7f8de40035b0 con 0x7f8df4073f90 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df40cfb00 msgr2=0x7f8df40cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df40cfb00 0x7f8df40cff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 >> v1:172.21.15.141:6789/0 conn(0x7f8df406dc20 legacy=0x7f8df415f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df4160610 con 0x7f8df4073f90 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f8de8001670 con 0x7f8df4073f90 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 >> v1:172.21.15.113:6789/0 conn(0x7f8df4073f90 legacy=0x7f8df4074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de4003a30 0x7f8de4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de4004850 0x7f8de4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df40cfb00 0x7f8de4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df41608f0 con 0x7f8de4004850 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de400bac0 con 0x7f8de4003a30 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de400bc80 con 0x7f8df40cfb00 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df3fff700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de4004850 0x7f8de4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df8b03700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de4003a30 0x7f8de4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.117+0000 7f8df9304700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df40cfb00 0x7f8de4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8df3fff700 1 -- 172.21.15.113:0/1160681840 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df40cfb00 msgr2=0x7f8de4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8df3fff700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df40cfb00 0x7f8de4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8df3fff700 1 -- 172.21.15.113:0/1160681840 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de4003a30 msgr2=0x7f8de4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8df3fff700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de4003a30 0x7f8de4003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8df3fff700 1 -- 172.21.15.113:0/1160681840 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de40035b0 con 0x7f8de4004850 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8df8b03700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de4003a30 0x7f8de4003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8df3fff700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de4004850 0x7f8de4004cd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8de0007fd0 tx=0x7f8de4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad2cf682115dea8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de00012f0 con 0x7f8de4004850 2024-01-15T11:25:36.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de00052f0 con 0x7f8de4004850 2024-01-15T11:25:36.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8df37fe700 1 -- 172.21.15.113:0/1160681840 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0006860 con 0x7f8de4004850 2024-01-15T11:25:36.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8dfad67700 1 -- 172.21.15.113:0/1160681840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de4004850 msgr2=0x7f8de4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:36.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8dfad67700 1 --2- 172.21.15.113:0/1160681840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de4004850 0x7f8de4004cd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8de0007fd0 tx=0x7f8de4005880 comp rx=0 tx=0).stop 2024-01-15T11:25:36.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8df9304700 1 -- 172.21.15.113:0/1160681840 reap_dead start 2024-01-15T11:25:36.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8dfad67700 1 -- 172.21.15.113:0/1160681840 shutdown_connections 2024-01-15T11:25:36.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8dfad67700 1 -- 172.21.15.113:0/1160681840 >> 172.21.15.113:0/1160681840 conn(0x7f8df40b3130 msgr2=0x7f8df40d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:36.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8dfad67700 1 -- 172.21.15.113:0/1160681840 shutdown_connections 2024-01-15T11:25:36.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.118+0000 7f8dfad67700 1 -- 172.21.15.113:0/1160681840 wait complete. 2024-01-15T11:25:36.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.120+0000 7f8dfad67700 1 Processor -- start 2024-01-15T11:25:36.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.120+0000 7f8dfad67700 1 -- start start 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.120+0000 7f8dfad67700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de4004850 0x7f8df415ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.120+0000 7f8dfad67700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df415d830 0x7f8df415b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.120+0000 7f8dfad67700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df40b5560 0x7f8df40af170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.120+0000 7f8dfad67700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df40cf6e0 con 0x7f8de4004850 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.120+0000 7f8dfad67700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df415dcb0 con 0x7f8df40b5560 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.120+0000 7f8dfad67700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df415b2f0 con 0x7f8df415d830 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df8b03700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de4004850 0x7f8df415ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df8b03700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de4004850 0x7f8df415ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59184/0 (socket says 172.21.15.113:59184) 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df8b03700 1 -- 172.21.15.113:0/2888460374 learned_addr learned my addr 172.21.15.113:0/2888460374 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df3fff700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df415d830 0x7f8df415b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df9304700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df40b5560 0x7f8df40af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df8b03700 1 -- 172.21.15.113:0/2888460374 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df415d830 msgr2=0x7f8df415b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df8b03700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df415d830 0x7f8df415b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:36.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df8b03700 1 -- 172.21.15.113:0/2888460374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df40b5560 msgr2=0x7f8df40af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df8b03700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df40b5560 0x7f8df40af170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df8b03700 1 -- 172.21.15.113:0/2888460374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df415dea0 con 0x7f8de4004850 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df9304700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df40b5560 0x7f8df40af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df8b03700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de4004850 0x7f8df415ef40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8de8000ff0 tx=0x7f8de800ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df17fa700 1 -- 172.21.15.113:0/2888460374 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8012590 con 0x7f8de4004850 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df17fa700 1 -- 172.21.15.113:0/2888460374 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de80126f0 con 0x7f8de4004850 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8df17fa700 1 -- 172.21.15.113:0/2888460374 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8012a60 con 0x7f8de4004850 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8dfad67700 1 -- 172.21.15.113:0/2888460374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df415e180 con 0x7f8de4004850 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.121+0000 7f8dfad67700 1 -- 172.21.15.113:0/2888460374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df415e740 con 0x7f8de4004850 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.122+0000 7f8dfad67700 1 -- 172.21.15.113:0/2888460374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df4067f60 con 0x7f8de4004850 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.123+0000 7f8df17fa700 1 -- 172.21.15.113:0/2888460374 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8de8012bc0 con 0x7f8de4004850 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.123+0000 7f8df17fa700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8dd806b350 0x7f8dd806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.123+0000 7f8df17fa700 1 -- 172.21.15.113:0/2888460374 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8de80897d0 con 0x7f8de4004850 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.123+0000 7f8df3fff700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8dd806b350 0x7f8dd806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:36.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.123+0000 7f8df3fff700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8dd806b350 0x7f8dd806d810 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8df413a740 tx=0x7f8de0011000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:36.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.127+0000 7f8df17fa700 1 -- 172.21.15.113:0/2888460374 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8de8054640 con 0x7f8de4004850 2024-01-15T11:25:36.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.361+0000 7f8dfad67700 1 -- 172.21.15.113:0/2888460374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8df40d1a10 con 0x7f8de4004850 2024-01-15T11:25:36.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.362+0000 7f8df17fa700 1 -- 172.21.15.113:0/2888460374 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8de8057c60 con 0x7f8de4004850 2024-01-15T11:25:36.361 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T11:25:36.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.364+0000 7f8dfad67700 1 -- 172.21.15.113:0/2888460374 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8dd806b350 msgr2=0x7f8dd806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:36.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.364+0000 7f8dfad67700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8dd806b350 0x7f8dd806d810 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8df413a740 tx=0x7f8de0011000 comp rx=0 tx=0).stop 2024-01-15T11:25:36.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.364+0000 7f8dfad67700 1 -- 172.21.15.113:0/2888460374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de4004850 msgr2=0x7f8df415ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:36.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.364+0000 7f8dfad67700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de4004850 0x7f8df415ef40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8de8000ff0 tx=0x7f8de800ea60 comp rx=0 tx=0).stop 2024-01-15T11:25:36.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.364+0000 7f8dfad67700 1 -- 172.21.15.113:0/2888460374 shutdown_connections 2024-01-15T11:25:36.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.364+0000 7f8dfad67700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8dd806b350 0x7f8dd806d810 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:36.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.364+0000 7f8dfad67700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df40b5560 0x7f8df40af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:36.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.364+0000 7f8dfad67700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de4004850 0x7f8df415ef40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:36.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.364+0000 7f8dfad67700 1 --2- 172.21.15.113:0/2888460374 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8df415d830 0x7f8df415b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:36.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.364+0000 7f8dfad67700 1 -- 172.21.15.113:0/2888460374 >> 172.21.15.113:0/2888460374 conn(0x7f8df40b3130 msgr2=0x7f8df40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:36.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.364+0000 7f8dfad67700 1 -- 172.21.15.113:0/2888460374 shutdown_connections 2024-01-15T11:25:36.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:36.364+0000 7f8dfad67700 1 -- 172.21.15.113:0/2888460374 wait complete. 2024-01-15T11:25:36.375 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T11:25:36.375 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T11:25:36.375 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi141.front.sepia.ceph.com 2024-01-15T11:25:36.375 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T11:25:36.375 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T11:25:36.375 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T11:25:36.376 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T11:25:36.376 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T11:25:36.376 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T11:25:36.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:36.376 DEBUG:teuthology.orchestra.run.smithi141:> ip addr 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:26 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f0 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:27 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f1 2024-01-15T11:25:36.399 INFO:teuthology.orchestra.run.smithi141.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:25:36.400 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6c brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:36.400 INFO:teuthology.orchestra.run.smithi141.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T11:25:36.400 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6d brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:36.400 INFO:teuthology.orchestra.run.smithi141.stdout: inet 172.21.15.141/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T11:25:36.400 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft 42458sec preferred_lft 42458sec 2024-01-15T11:25:36.400 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::91fc:8187:19e2:bf82/64 scope link noprefixroute 2024-01-15T11:25:36.400 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:36.400 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-15T11:25:36.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:36.400 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-15T11:25:36.401 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-15T11:25:36.401 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link add name ceph-brx type bridge 2024-01-15T11:25:36.401 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip addr flush dev ceph-brx 2024-01-15T11:25:36.401 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set ceph-brx up 2024-01-15T11:25:36.401 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-15T11:25:36.401 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-15T11:25:36.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:36.582 DEBUG:teuthology.orchestra.run.smithi141:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T11:25:36.610 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-15T11:25:36.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:36.611 DEBUG:teuthology.orchestra.run.smithi141:> route 2024-01-15T11:25:36.684 INFO:teuthology.orchestra.run.smithi141.stdout:Kernel IP routing table 2024-01-15T11:25:36.684 INFO:teuthology.orchestra.run.smithi141.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T11:25:36.684 INFO:teuthology.orchestra.run.smithi141.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T11:25:36.684 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T11:25:36.684 INFO:teuthology.orchestra.run.smithi141.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T11:25:36.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:36.685 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-15T11:25:36.685 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-15T11:25:36.685 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T11:25:36.685 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T11:25:36.685 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T11:25:36.685 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-15T11:25:36.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:36.911 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-15T11:25:36.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:36.928 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list-id 2024-01-15T11:25:36.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:36.985 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-15T11:25:36.985 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-15T11:25:36.986 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T11:25:36.986 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-15T11:25:36.986 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-15T11:25:37.080 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-15T11:25:37.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:37.080 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-15T11:25:37.081 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-15T11:25:37.081 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-15T11:25:37.081 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-15T11:25:37.081 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-15T11:25:37.081 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-15T11:25:37.081 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-15T11:25:37.081 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-15T11:25:37.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:37.287 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-15T11:25:37.287 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-15T11:25:37.287 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set brx.0 up 2024-01-15T11:25:37.288 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set dev brx.0 master ceph-brx 2024-01-15T11:25:37.288 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-15T11:25:37.385 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T11:25:37.385 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T11:25:37.401 INFO:teuthology.orchestra.run.smithi141.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T11:25:37.402 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T11:25:37.402 DEBUG:teuthology.orchestra.run.smithi141:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T11:25:37.458 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-15T11:25:37.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:37.458 DEBUG:teuthology.orchestra.run.smithi141:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-15T11:25:37.545 INFO:teuthology.orchestra.run.smithi141.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-15T11:25:37.545 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-15T11:25:37.545 INFO:teuthology.orchestra.run.smithi141.stdout:invalid new device string format 2024-01-15T11:25:37.625 INFO:teuthology.orchestra.run.smithi141.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-15T11:25:37.625 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-15T11:25:37.625 INFO:teuthology.orchestra.run.smithi141.stdout:invalid new device string format 2024-01-15T11:25:37.625 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.141:3300,172.21.15.141:3301" 2024-01-15T11:25:37.625 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T11:25:37.643 INFO:teuthology.orchestra.run.smithi141.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-15T11:25:37.643 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-15T11:25:37.643 INFO:teuthology.orchestra.run.smithi141.stdout:invalid new device string format 2024-01-15T11:25:37.643 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.141:3300,172.21.15.141:3301" 2024-01-15T11:25:37.643 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T11:25:37.643 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: resolved to: "172.21.15.113:6789,172.21.15.141:6789,172.21.15.141:6790" 2024-01-15T11:25:37.644 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.113:6789,172.21.15.141:6789,172.21.15.141:6790:/ 2024-01-15T11:25:37.644 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=910f8628-cd75-4a37-bf4e-e3603a95290d" will pass to kernel 2024-01-15T11:25:37.646 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-15T11:25:37.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:37.646 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T11:25:37.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:37.673 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:25:37.746 INFO:teuthology.orchestra.run.smithi141.stdout:client4600 2024-01-15T11:25:37.746 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:25:37.746 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:37.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:37.804 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:25:37.876 INFO:teuthology.orchestra.run.smithi141.stdout:client4600 2024-01-15T11:25:37.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:37.877 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:25:37.948 INFO:teuthology.orchestra.run.smithi141.stdout:client4600 2024-01-15T11:25:37.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:37.949 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/910f8628-cd75-4a37-bf4e-e3603a95290d.client4600/status') 2024-01-15T11:25:38.028 INFO:teuthology.orchestra.run.smithi141.stdout:instance: client.4600 (3)192.168.0.1:0/2753762582 2024-01-15T11:25:38.028 INFO:teuthology.orchestra.run.smithi141.stdout:blocklisted: false 2024-01-15T11:25:38.029 INFO:teuthology.orchestra.run.smithi141.stderr:0+1 records in 2024-01-15T11:25:38.029 INFO:teuthology.orchestra.run.smithi141.stderr:0+1 records out 2024-01-15T11:25:38.029 INFO:teuthology.orchestra.run.smithi141.stderr:69 bytes copied, 2.0231e-05 s, 3.4 MB/s 2024-01-15T11:25:38.030 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-01-15T11:25:38.030 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T11:25:38.030 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-15T11:25:38.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T11:25:38.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.166+0000 7fb6a1755700 1 Processor -- start 2024-01-15T11:25:38.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.166+0000 7fb6a1755700 1 -- start start 2024-01-15T11:25:38.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.166+0000 7fb6a1755700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb69c15e1e0 0x7fb69c0d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:38.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.166+0000 7fb6a1755700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c0d20b0 con 0x7fb69c0d0f80 2024-01-15T11:25:38.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.166+0000 7fb6a1755700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c0d2390 con 0x7fb69c0d3a00 2024-01-15T11:25:38.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.166+0000 7fb6a1755700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb69c0d3150 con 0x7fb69c15e1e0 2024-01-15T11:25:38.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb69b7fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb69c0d0f80 0x7fb69c0b6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42492/0 (socket says 172.21.15.113:42492) 2024-01-15T11:25:38.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb69b7fe700 1 -- 172.21.15.113:0/3198077219 learned_addr learned my addr 172.21.15.113:0/3198077219 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:38.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb69affd700 1 --2- 172.21.15.113:0/3198077219 >> v2:172.21.15.141:3300/0 conn(0x7fb69c15e1e0 0x7fb69c0d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379914296 0 0) 0x7fb69c0d20b0 con 0x7fb69c0d0f80 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6840035b0 con 0x7fb69c0d0f80 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245798928 0 0) 0x7fb69c0d2390 con 0x7fb69c0d3a00 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c0d20b0 con 0x7fb69c0d3a00 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2504365481 0 0) 0x7fb6840035b0 con 0x7fb69c0d0f80 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 >> v2:172.21.15.141:3300/0 conn(0x7fb69c15e1e0 msgr2=0x7fb69c0d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 --2- 172.21.15.113:0/3198077219 >> v2:172.21.15.141:3300/0 conn(0x7fb69c15e1e0 0x7fb69c0d34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 >> v1:172.21.15.141:6789/0 conn(0x7fb69c0d3a00 legacy=0x7fb69c0d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c0d2670 con 0x7fb69c0d0f80 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fb6940019f0 con 0x7fb69c0d0f80 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 >> v1:172.21.15.113:6789/0 conn(0x7fb69c0d0f80 legacy=0x7fb69c0b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 --2- 172.21.15.113:0/3198077219 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 0x7fb684003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:38.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 --2- 172.21.15.113:0/3198077219 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb684004850 0x7fb684004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 --2- 172.21.15.113:0/3198077219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb684005210 0x7fb68400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c0d3150 con 0x7fb684003be0 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68400c030 con 0x7fb684005210 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68400c1f0 con 0x7fb684004850 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb69affd700 1 --2- 172.21.15.113:0/3198077219 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 0x7fb684003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb69b7fe700 1 --2- 172.21.15.113:0/3198077219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb684005210 0x7fb68400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.167+0000 7fb69a7fc700 1 --2- 172.21.15.113:0/3198077219 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb684004850 0x7fb684004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb69affd700 1 -- 172.21.15.113:0/3198077219 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb684004850 msgr2=0x7fb684004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb69affd700 1 --2- 172.21.15.113:0/3198077219 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb684004850 0x7fb684004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb69affd700 1 -- 172.21.15.113:0/3198077219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb684005210 msgr2=0x7fb68400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb69affd700 1 --2- 172.21.15.113:0/3198077219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb684005210 0x7fb68400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb69b7fe700 1 -- 172.21.15.113:0/3198077219 reap_dead start 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb69affd700 1 -- 172.21.15.113:0/3198077219 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6840035b0 con 0x7fb684003be0 2024-01-15T11:25:38.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb69affd700 1 --2- 172.21.15.113:0/3198077219 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 0x7fb684003fe0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009b60 tx=0x7fb684005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ece4f899a82228 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:38.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c00e6c0 con 0x7fb684003be0 2024-01-15T11:25:38.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68c00e820 con 0x7fb684003be0 2024-01-15T11:25:38.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb699ffb700 1 -- 172.21.15.113:0/3198077219 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c00eb90 con 0x7fb684003be0 2024-01-15T11:25:38.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb6a1755700 1 -- 172.21.15.113:0/3198077219 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 msgr2=0x7fb684003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:38.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb6a1755700 1 --2- 172.21.15.113:0/3198077219 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 0x7fb684003fe0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009b60 tx=0x7fb684005880 comp rx=0 tx=0).stop 2024-01-15T11:25:38.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb6a1755700 1 -- 172.21.15.113:0/3198077219 shutdown_connections 2024-01-15T11:25:38.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb6a1755700 1 --2- 172.21.15.113:0/3198077219 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 0x7fb684003fe0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:38.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb6a1755700 1 -- 172.21.15.113:0/3198077219 >> 172.21.15.113:0/3198077219 conn(0x7fb69c0b5b50 msgr2=0x7fb69c0b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:38.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb6a1755700 1 -- 172.21.15.113:0/3198077219 shutdown_connections 2024-01-15T11:25:38.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.168+0000 7fb6a1755700 1 -- 172.21.15.113:0/3198077219 wait complete. 2024-01-15T11:25:38.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.170+0000 7fb6a1755700 1 Processor -- start 2024-01-15T11:25:38.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.170+0000 7fb6a1755700 1 -- start start 2024-01-15T11:25:38.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb6a1755700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 0x7fb69c0ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:38.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb6a1755700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb69c1564d0 0x7fb69c156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:38.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb6a1755700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb69c160910 0x7fb69c160da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:38.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb6a1755700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c1612e0 con 0x7fb684003be0 2024-01-15T11:25:38.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb6a1755700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c14c3f0 con 0x7fb69c160910 2024-01-15T11:25:38.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb6a1755700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c157020 con 0x7fb69c1564d0 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb69affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 0x7fb69c0ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb69affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 0x7fb69c0ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59208/0 (socket says 172.21.15.113:59208) 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb69affd700 1 -- 172.21.15.113:0/430087027 learned_addr learned my addr 172.21.15.113:0/430087027 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb69a7fc700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb69c1564d0 0x7fb69c156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb69b7fe700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb69c160910 0x7fb69c160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb69affd700 1 -- 172.21.15.113:0/430087027 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb69c1564d0 msgr2=0x7fb69c156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb69affd700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb69c1564d0 0x7fb69c156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb69affd700 1 -- 172.21.15.113:0/430087027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb69c160910 msgr2=0x7fb69c160da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb69affd700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb69c160910 0x7fb69c160da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb69affd700 1 -- 172.21.15.113:0/430087027 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c14bac0 con 0x7fb684003be0 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb69affd700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 0x7fb69c0ca4f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c000 tx=0x7fb68c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb6837fe700 1 -- 172.21.15.113:0/430087027 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c00e6c0 con 0x7fb684003be0 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb6837fe700 1 -- 172.21.15.113:0/430087027 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68c0038a0 con 0x7fb684003be0 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb6a1755700 1 -- 172.21.15.113:0/430087027 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69c14bda0 con 0x7fb684003be0 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb6837fe700 1 -- 172.21.15.113:0/430087027 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c003c50 con 0x7fb684003be0 2024-01-15T11:25:38.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.171+0000 7fb6a1755700 1 -- 172.21.15.113:0/430087027 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69c0caa30 con 0x7fb684003be0 2024-01-15T11:25:38.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.172+0000 7fb6a1755700 1 -- 172.21.15.113:0/430087027 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69c0cad30 con 0x7fb684003be0 2024-01-15T11:25:38.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.173+0000 7fb6837fe700 1 -- 172.21.15.113:0/430087027 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb68c013470 con 0x7fb684003be0 2024-01-15T11:25:38.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.173+0000 7fb6837fe700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb68806b350 0x7fb68806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:38.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.173+0000 7fb6837fe700 1 -- 172.21.15.113:0/430087027 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb68c08b380 con 0x7fb684003be0 2024-01-15T11:25:38.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.173+0000 7fb69a7fc700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb68806b350 0x7fb68806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:38.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.174+0000 7fb69a7fc700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb68806b350 0x7fb68806d810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb690000c00 tx=0x7fb690009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:38.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.178+0000 7fb6837fe700 1 -- 172.21.15.113:0/430087027 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb68c0563a0 con 0x7fb684003be0 2024-01-15T11:25:38.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.416+0000 7fb6a1755700 1 -- 172.21.15.113:0/430087027 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb69c0751e0 con 0x7fb684003be0 2024-01-15T11:25:38.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.416+0000 7fb6837fe700 1 -- 172.21.15.113:0/430087027 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb68c0599c0 con 0x7fb684003be0 2024-01-15T11:25:38.416 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T11:25:38.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.418+0000 7fb6a1755700 1 -- 172.21.15.113:0/430087027 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb68806b350 msgr2=0x7fb68806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:38.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.418+0000 7fb6a1755700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb68806b350 0x7fb68806d810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb690000c00 tx=0x7fb690009040 comp rx=0 tx=0).stop 2024-01-15T11:25:38.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.418+0000 7fb6a1755700 1 -- 172.21.15.113:0/430087027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 msgr2=0x7fb69c0ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:38.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.418+0000 7fb6a1755700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 0x7fb69c0ca4f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c000 tx=0x7fb68c0047c0 comp rx=0 tx=0).stop 2024-01-15T11:25:38.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.418+0000 7fb6a1755700 1 -- 172.21.15.113:0/430087027 shutdown_connections 2024-01-15T11:25:38.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.418+0000 7fb6a1755700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb68806b350 0x7fb68806d810 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:38.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.418+0000 7fb6a1755700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb69c160910 0x7fb69c160da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:38.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.418+0000 7fb6a1755700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb684003be0 0x7fb69c0ca4f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:38.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.418+0000 7fb6a1755700 1 --2- 172.21.15.113:0/430087027 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb69c1564d0 0x7fb69c156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:38.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.418+0000 7fb6a1755700 1 -- 172.21.15.113:0/430087027 >> 172.21.15.113:0/430087027 conn(0x7fb69c0b5b50 msgr2=0x7fb69c0b7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:38.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.419+0000 7fb6a1755700 1 -- 172.21.15.113:0/430087027 shutdown_connections 2024-01-15T11:25:38.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:38.419+0000 7fb6a1755700 1 -- 172.21.15.113:0/430087027 wait complete. 2024-01-15T11:25:38.429 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T11:25:38.430 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T11:25:38.430 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi141.front.sepia.ceph.com 2024-01-15T11:25:38.430 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-01-15T11:25:38.430 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-15T11:25:38.430 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T11:25:38.430 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T11:25:38.430 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T11:25:38.430 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-01-15T11:25:38.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:38.431 DEBUG:teuthology.orchestra.run.smithi141:> ip addr 2024-01-15T11:25:38.449 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T11:25:38.449 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T11:25:38.449 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T11:25:38.449 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.449 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-15T11:25:38.449 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.449 INFO:teuthology.orchestra.run.smithi141.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:25:38.449 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:26 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:38.449 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f0 2024-01-15T11:25:38.449 INFO:teuthology.orchestra.run.smithi141.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:27 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f1 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6c brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6d brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout: inet 172.21.15.141/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft 42456sec preferred_lft 42456sec 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::91fc:8187:19e2:bf82/64 scope link noprefixroute 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 1a:e8:62:f1:ab:25 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:25:38.450 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T11:25:38.451 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.451 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::cc20:9bff:fea4:a969/64 scope link 2024-01-15T11:25:38.451 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.451 INFO:teuthology.orchestra.run.smithi141.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T11:25:38.451 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 1a:e8:62:f1:ab:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T11:25:38.451 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::18e8:62ff:fef1:ab25/64 scope link tentative 2024-01-15T11:25:38.451 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:38.452 DEBUG:teuthology.orchestra.run.smithi141:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T11:25:38.515 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-15T11:25:38.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:38.517 DEBUG:teuthology.orchestra.run.smithi141:> route 2024-01-15T11:25:38.575 INFO:teuthology.orchestra.run.smithi141.stdout:Kernel IP routing table 2024-01-15T11:25:38.575 INFO:teuthology.orchestra.run.smithi141.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T11:25:38.575 INFO:teuthology.orchestra.run.smithi141.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T11:25:38.575 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T11:25:38.575 INFO:teuthology.orchestra.run.smithi141.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T11:25:38.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:38.576 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-15T11:25:38.576 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-15T11:25:38.576 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T11:25:38.576 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T11:25:38.576 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T11:25:38.576 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-15T11:25:38.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:38.681 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-15T11:25:38.698 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T11:25:38.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:38.698 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list-id 2024-01-15T11:25:38.754 INFO:teuthology.orchestra.run.smithi141.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-01-15T11:25:38.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:38.755 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-15T11:25:38.755 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-15T11:25:38.755 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T11:25:38.755 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-01-15T11:25:38.755 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-15T11:25:38.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:38.849 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-15T11:25:38.883 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T11:25:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T11:25:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T11:25:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-15T11:25:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.884 INFO:teuthology.orchestra.run.smithi141.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T11:25:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 76:24:74:4a:1f:66 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T11:25:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T11:25:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::7424:74ff:fe4a:1f66/64 scope link 2024-01-15T11:25:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:38.901 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-15T11:25:38.981 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T11:25:38.982 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T11:25:38.982 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T11:25:38.982 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.982 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-15T11:25:38.982 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.982 INFO:teuthology.orchestra.run.smithi141.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T11:25:38.982 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 76:24:74:4a:1f:66 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T11:25:38.982 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T11:25:38.982 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.982 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::7424:74ff:fe4a:1f66/64 scope link 2024-01-15T11:25:38.982 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:25:38.995 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-01-15T11:25:38.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:38.995 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-15T11:25:38.995 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-15T11:25:38.995 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-01-15T11:25:38.995 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-15T11:25:38.995 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-01-15T11:25:38.996 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-01-15T11:25:38.996 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-01-15T11:25:38.996 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-15T11:25:39.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:39.217 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-15T11:25:39.217 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-15T11:25:39.217 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set brx.1 up 2024-01-15T11:25:39.217 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set dev brx.1 master ceph-brx 2024-01-15T11:25:39.217 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-15T11:25:39.315 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T11:25:39.315 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-01-15T11:25:39.332 INFO:teuthology.orchestra.run.smithi141.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-01-15T11:25:39.333 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T11:25:39.333 DEBUG:teuthology.orchestra.run.smithi141:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-01-15T11:25:39.389 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-15T11:25:39.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:39.390 DEBUG:teuthology.orchestra.run.smithi141:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-01-15T11:25:39.475 INFO:teuthology.orchestra.run.smithi141.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-01-15T11:25:39.476 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-01-15T11:25:39.476 INFO:teuthology.orchestra.run.smithi141.stdout:invalid new device string format 2024-01-15T11:25:39.485 INFO:teuthology.orchestra.run.smithi141.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-01-15T11:25:39.485 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-01-15T11:25:39.485 INFO:teuthology.orchestra.run.smithi141.stdout:invalid new device string format 2024-01-15T11:25:39.485 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.141:3300,172.21.15.141:3301" 2024-01-15T11:25:39.485 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T11:25:39.503 INFO:teuthology.orchestra.run.smithi141.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-01-15T11:25:39.503 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-01-15T11:25:39.503 INFO:teuthology.orchestra.run.smithi141.stdout:invalid new device string format 2024-01-15T11:25:39.503 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.141:3300,172.21.15.141:3301" 2024-01-15T11:25:39.504 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T11:25:39.504 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: resolved to: "172.21.15.113:6789,172.21.15.141:6789,172.21.15.141:6790" 2024-01-15T11:25:39.504 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.113:6789,172.21.15.141:6789,172.21.15.141:6790:/ 2024-01-15T11:25:39.504 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=910f8628-cd75-4a37-bf4e-e3603a95290d" will pass to kernel 2024-01-15T11:25:39.506 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-15T11:25:39.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:39.506 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-01-15T11:25:39.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:39.534 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:25:39.606 INFO:teuthology.orchestra.run.smithi141.stdout:client4481 2024-01-15T11:25:39.606 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:25:39.607 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:39.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:39.665 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:25:39.737 INFO:teuthology.orchestra.run.smithi141.stdout:client4481 2024-01-15T11:25:39.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:39.737 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:25:39.810 INFO:teuthology.orchestra.run.smithi141.stdout:client4481 2024-01-15T11:25:39.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:39.811 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/910f8628-cd75-4a37-bf4e-e3603a95290d.client4481/status') 2024-01-15T11:25:39.891 INFO:teuthology.orchestra.run.smithi141.stdout:instance: client.4481 (3)192.168.0.2:0/3494028003 2024-01-15T11:25:39.891 INFO:teuthology.orchestra.run.smithi141.stdout:blocklisted: false 2024-01-15T11:25:39.891 INFO:teuthology.orchestra.run.smithi141.stderr:0+1 records in 2024-01-15T11:25:39.892 INFO:teuthology.orchestra.run.smithi141.stderr:0+1 records out 2024-01-15T11:25:39.892 INFO:teuthology.orchestra.run.smithi141.stderr:69 bytes copied, 2.0153e-05 s, 3.4 MB/s 2024-01-15T11:25:39.893 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-01-15T11:25:39.983 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:40.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.118+0000 7eff03e49700 1 Processor -- start 2024-01-15T11:25:40.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.119+0000 7eff03e49700 1 -- start start 2024-01-15T11:25:40.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff03e49700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7efefc0b50c0 0x7efefc0cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff03e49700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc06bf50 con 0x7efefc06f460 2024-01-15T11:25:40.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff03e49700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc15b030 con 0x7efefc15ac50 2024-01-15T11:25:40.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff03e49700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc06e810 con 0x7efefc0b50c0 2024-01-15T11:25:40.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff013e4700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7efefc06f460 0x7efefc0b49b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42498/0 (socket says 172.21.15.113:42498) 2024-01-15T11:25:40.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff013e4700 1 -- 172.21.15.113:0/3219697206 learned_addr learned my addr 172.21.15.113:0/3219697206 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:40.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410811844 0 0) 0x7efefc06bf50 con 0x7efefc06f460 2024-01-15T11:25:40.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff023e6700 1 --2- 172.21.15.113:0/3219697206 >> v2:172.21.15.141:3300/0 conn(0x7efefc0b50c0 0x7efefc0cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee80035b0 con 0x7efefc06f460 2024-01-15T11:25:40.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2387635085 0 0) 0x7efee80035b0 con 0x7efefc06f460 2024-01-15T11:25:40.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 >> v2:172.21.15.141:3300/0 conn(0x7efefc0b50c0 msgr2=0x7efefc0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:40.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 --2- 172.21.15.113:0/3219697206 >> v2:172.21.15.141:3300/0 conn(0x7efefc0b50c0 0x7efefc0cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 >> v1:172.21.15.141:6789/0 conn(0x7efefc15ac50 legacy=0x7efefc06c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc06b340 con 0x7efefc06f460 2024-01-15T11:25:40.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7efef8001670 con 0x7efefc06f460 2024-01-15T11:25:40.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 >> v1:172.21.15.113:6789/0 conn(0x7efefc06f460 legacy=0x7efefc0b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 --2- 172.21.15.113:0/3219697206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efee8003a00 0x7efee8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 --2- 172.21.15.113:0/3219697206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efee8004400 0x7efee80048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 --2- 172.21.15.113:0/3219697206 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efee80050d0 0x7efee800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc06e810 con 0x7efee8003a00 2024-01-15T11:25:40.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee800c030 con 0x7efee8004400 2024-01-15T11:25:40.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee800c1f0 con 0x7efee80050d0 2024-01-15T11:25:40.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.120+0000 7eff01be5700 1 --2- 172.21.15.113:0/3219697206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efee8003a00 0x7efee8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.121+0000 7eff013e4700 1 --2- 172.21.15.113:0/3219697206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efee8004400 0x7efee80048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.121+0000 7eff023e6700 1 --2- 172.21.15.113:0/3219697206 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efee80050d0 0x7efee800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.121+0000 7eff01be5700 1 -- 172.21.15.113:0/3219697206 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efee80050d0 msgr2=0x7efee800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:40.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.121+0000 7eff01be5700 1 --2- 172.21.15.113:0/3219697206 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efee80050d0 0x7efee800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.121+0000 7eff01be5700 1 -- 172.21.15.113:0/3219697206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efee8004400 msgr2=0x7efee80048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:40.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.121+0000 7eff01be5700 1 --2- 172.21.15.113:0/3219697206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efee8004400 0x7efee80048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.121+0000 7eff01be5700 1 -- 172.21.15.113:0/3219697206 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee80035b0 con 0x7efee8003a00 2024-01-15T11:25:40.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.121+0000 7eff023e6700 1 -- 172.21.15.113:0/3219697206 reap_dead start 2024-01-15T11:25:40.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.121+0000 7eff01be5700 1 --2- 172.21.15.113:0/3219697206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efee8003a00 0x7efee8003de0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efef00018d0 tx=0x7efee8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9868d932b576499f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:40.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.121+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef00052d0 con 0x7efee8003a00 2024-01-15T11:25:40.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.121+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef0005430 con 0x7efee8003a00 2024-01-15T11:25:40.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.121+0000 7eff00be3700 1 -- 172.21.15.113:0/3219697206 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef00038c0 con 0x7efee8003a00 2024-01-15T11:25:40.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.122+0000 7eff03e49700 1 -- 172.21.15.113:0/3219697206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efee8003a00 msgr2=0x7efee8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.122+0000 7eff03e49700 1 --2- 172.21.15.113:0/3219697206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efee8003a00 0x7efee8003de0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efef00018d0 tx=0x7efee8005880 comp rx=0 tx=0).stop 2024-01-15T11:25:40.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.122+0000 7eff03e49700 1 -- 172.21.15.113:0/3219697206 shutdown_connections 2024-01-15T11:25:40.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.122+0000 7eff03e49700 1 --2- 172.21.15.113:0/3219697206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efee8003a00 0x7efee8003de0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.122+0000 7eff03e49700 1 -- 172.21.15.113:0/3219697206 >> 172.21.15.113:0/3219697206 conn(0x7efefc0b3130 msgr2=0x7efefc0d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:40.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.122+0000 7eff03e49700 1 -- 172.21.15.113:0/3219697206 shutdown_connections 2024-01-15T11:25:40.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.122+0000 7eff03e49700 1 -- 172.21.15.113:0/3219697206 wait complete. 2024-01-15T11:25:40.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.123+0000 7eff03e49700 1 Processor -- start 2024-01-15T11:25:40.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.123+0000 7eff03e49700 1 -- start start 2024-01-15T11:25:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.124+0000 7eff03e49700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efee8003a00 0x7efefc1f0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.124+0000 7eff03e49700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc154730 0x7efefc1572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.124+0000 7eff03e49700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efefc071c00 0x7efefc072080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.124+0000 7eff03e49700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0725c0 con 0x7efefc154730 2024-01-15T11:25:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.124+0000 7eff03e49700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0aa350 con 0x7efee8003a00 2024-01-15T11:25:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.124+0000 7eff03e49700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc154bb0 con 0x7efefc071c00 2024-01-15T11:25:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.124+0000 7eff013e4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc154730 0x7efefc1572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.124+0000 7eff013e4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc154730 0x7efefc1572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59230/0 (socket says 172.21.15.113:59230) 2024-01-15T11:25:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.124+0000 7eff013e4700 1 -- 172.21.15.113:0/3003083497 learned_addr learned my addr 172.21.15.113:0/3003083497 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.124+0000 7eff01be5700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efee8003a00 0x7efefc1f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.124+0000 7eff023e6700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efefc071c00 0x7efefc072080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.125+0000 7eff013e4700 1 -- 172.21.15.113:0/3003083497 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efefc071c00 msgr2=0x7efefc072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.125+0000 7eff013e4700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efefc071c00 0x7efefc072080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.125+0000 7eff013e4700 1 -- 172.21.15.113:0/3003083497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efee8003a00 msgr2=0x7efefc1f0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.125+0000 7eff013e4700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efee8003a00 0x7efefc1f0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.125+0000 7eff013e4700 1 -- 172.21.15.113:0/3003083497 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc0a9460 con 0x7efefc154730 2024-01-15T11:25:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.125+0000 7eff013e4700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc154730 0x7efefc1572c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efef80072c0 tx=0x7efef80075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.125+0000 7efeee7fc700 1 -- 172.21.15.113:0/3003083497 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8011580 con 0x7efefc154730 2024-01-15T11:25:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.125+0000 7efeee7fc700 1 -- 172.21.15.113:0/3003083497 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef8011720 con 0x7efefc154730 2024-01-15T11:25:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.125+0000 7efeee7fc700 1 -- 172.21.15.113:0/3003083497 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8011ad0 con 0x7efefc154730 2024-01-15T11:25:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.125+0000 7eff03e49700 1 -- 172.21.15.113:0/3003083497 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efefc0a9740 con 0x7efefc154730 2024-01-15T11:25:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.125+0000 7eff03e49700 1 -- 172.21.15.113:0/3003083497 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc0a9d00 con 0x7efefc154730 2024-01-15T11:25:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.126+0000 7eff03e49700 1 -- 172.21.15.113:0/3003083497 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efefc067f60 con 0x7efefc154730 2024-01-15T11:25:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.126+0000 7efeee7fc700 1 -- 172.21.15.113:0/3003083497 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efef8011c30 con 0x7efefc154730 2024-01-15T11:25:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.127+0000 7efeee7fc700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efee006b350 0x7efee006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.127+0000 7efeee7fc700 1 -- 172.21.15.113:0/3003083497 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efef80899d0 con 0x7efefc154730 2024-01-15T11:25:40.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.127+0000 7eff01be5700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efee006b350 0x7efee006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.127+0000 7eff01be5700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efee006b350 0x7efee006d810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efef0015040 tx=0x7efef0005000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:40.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.131+0000 7efeee7fc700 1 -- 172.21.15.113:0/3003083497 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efef80548d0 con 0x7efefc154730 2024-01-15T11:25:40.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.327+0000 7eff03e49700 1 -- 172.21.15.113:0/3003083497 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efefc06eb60 con 0x7efefc154730 2024-01-15T11:25:40.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.328+0000 7efeee7fc700 1 -- 172.21.15.113:0/3003083497 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7efef8057ef0 con 0x7efefc154730 2024-01-15T11:25:40.327 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:40.327 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:40.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.330+0000 7eff03e49700 1 -- 172.21.15.113:0/3003083497 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efee006b350 msgr2=0x7efee006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.330+0000 7eff03e49700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efee006b350 0x7efee006d810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efef0015040 tx=0x7efef0005000 comp rx=0 tx=0).stop 2024-01-15T11:25:40.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.330+0000 7eff03e49700 1 -- 172.21.15.113:0/3003083497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc154730 msgr2=0x7efefc1572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.330+0000 7eff03e49700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc154730 0x7efefc1572c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efef80072c0 tx=0x7efef80075d0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.330+0000 7eff03e49700 1 -- 172.21.15.113:0/3003083497 shutdown_connections 2024-01-15T11:25:40.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.330+0000 7eff03e49700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efee006b350 0x7efee006d810 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.330+0000 7eff03e49700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efefc071c00 0x7efefc072080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.330+0000 7eff03e49700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efee8003a00 0x7efefc1f0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.330+0000 7eff03e49700 1 --2- 172.21.15.113:0/3003083497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc154730 0x7efefc1572c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.330+0000 7eff03e49700 1 -- 172.21.15.113:0/3003083497 >> 172.21.15.113:0/3003083497 conn(0x7efefc0b3130 msgr2=0x7efefc0cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:40.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.330+0000 7eff03e49700 1 -- 172.21.15.113:0/3003083497 shutdown_connections 2024-01-15T11:25:40.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.330+0000 7eff03e49700 1 -- 172.21.15.113:0/3003083497 wait complete. 2024-01-15T11:25:40.342 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T11:25:13.398220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T11:25:14.589439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:40.342 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:40.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.468+0000 7f108f58f700 1 Processor -- start 2024-01-15T11:25:40.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.469+0000 7f108f58f700 1 -- start start 2024-01-15T11:25:40.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.469+0000 7f108f58f700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f108815e110 0x7f108806f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.469+0000 7f108f58f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f10880afb30 0x7f108806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.469+0000 7f108f58f700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10880affe0 con 0x7f108806b530 2024-01-15T11:25:40.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.469+0000 7f108f58f700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f108806d980 con 0x7f10880afb30 2024-01-15T11:25:40.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.469+0000 7f108f58f700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10880b0260 con 0x7f108815e110 2024-01-15T11:25:40.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f108cb2a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f10880afb30 0x7f108806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f108db2c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f108806b530 0x7f108806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42510/0 (socket says 172.21.15.113:42510) 2024-01-15T11:25:40.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f108db2c700 1 -- 172.21.15.113:0/381354772 learned_addr learned my addr 172.21.15.113:0/381354772 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:40.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f108d32b700 1 --2- 172.21.15.113:0/381354772 >> v2:172.21.15.141:3300/0 conn(0x7f108815e110 0x7f108806f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f108cb2a700 1 -- 172.21.15.113:0/381354772 >> v2:172.21.15.141:3300/0 conn(0x7f108815e110 msgr2=0x7f108806f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:40.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f108cb2a700 1 --2- 172.21.15.113:0/381354772 >> v2:172.21.15.141:3300/0 conn(0x7f108815e110 0x7f108806f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f108cb2a700 1 -- 172.21.15.113:0/381354772 >> v1:172.21.15.113:6789/0 conn(0x7f108806b530 legacy=0x7f108806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f108cb2a700 1 -- 172.21.15.113:0/381354772 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10880b0540 con 0x7f10880afb30 2024-01-15T11:25:40.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f107ffff700 1 -- 172.21.15.113:0/381354772 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486164561 0 0) 0x7f10880affe0 con 0x7f108806b530 2024-01-15T11:25:40.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f108cb2a700 1 --2- 172.21.15.113:0/381354772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10880afb30 0x7f108806aff0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1088073030 tx=0x7f1084009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b174326941655236 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:40.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f107ffff700 1 -- 172.21.15.113:0/381354772 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084011830 con 0x7f10880afb30 2024-01-15T11:25:40.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f107ffff700 1 -- 172.21.15.113:0/381354772 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1084011990 con 0x7f10880afb30 2024-01-15T11:25:40.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.470+0000 7f107ffff700 1 -- 172.21.15.113:0/381354772 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084011d20 con 0x7f10880afb30 2024-01-15T11:25:40.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.471+0000 7f108f58f700 1 -- 172.21.15.113:0/381354772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10880afb30 msgr2=0x7f108806aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.471+0000 7f108f58f700 1 --2- 172.21.15.113:0/381354772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10880afb30 0x7f108806aff0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1088073030 tx=0x7f1084009b20 comp rx=0 tx=0).stop 2024-01-15T11:25:40.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.471+0000 7f108f58f700 1 -- 172.21.15.113:0/381354772 shutdown_connections 2024-01-15T11:25:40.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.471+0000 7f108f58f700 1 --2- 172.21.15.113:0/381354772 >> v2:172.21.15.141:3300/0 conn(0x7f108815e110 0x7f108806f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.471+0000 7f108f58f700 1 --2- 172.21.15.113:0/381354772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10880afb30 0x7f108806aff0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.471+0000 7f108f58f700 1 -- 172.21.15.113:0/381354772 >> 172.21.15.113:0/381354772 conn(0x7f10880b3250 msgr2=0x7f10880d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:40.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.471+0000 7f108f58f700 1 -- 172.21.15.113:0/381354772 shutdown_connections 2024-01-15T11:25:40.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.471+0000 7f108f58f700 1 -- 172.21.15.113:0/381354772 wait complete. 2024-01-15T11:25:40.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.472+0000 7f108f58f700 1 Processor -- start 2024-01-15T11:25:40.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.472+0000 7f108f58f700 1 -- start start 2024-01-15T11:25:40.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.473+0000 7f108f58f700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f108806b530 0x7f108814d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.473+0000 7f108f58f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10880afb30 0x7f10881f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.473+0000 7f108f58f700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f108815e110 0x7f10880cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.473+0000 7f108f58f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10881f47a0 con 0x7f10880afb30 2024-01-15T11:25:40.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.473+0000 7f108f58f700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108814e500 con 0x7f108806b530 2024-01-15T11:25:40.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.473+0000 7f108f58f700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f108814dc60 con 0x7f108815e110 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.473+0000 7f108cb2a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10880afb30 0x7f10881f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.473+0000 7f108cb2a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10880afb30 0x7f10881f4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59242/0 (socket says 172.21.15.113:59242) 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.473+0000 7f108cb2a700 1 -- 172.21.15.113:0/334544115 learned_addr learned my addr 172.21.15.113:0/334544115 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f108d32b700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f108806b530 0x7f108814d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f108cb2a700 1 -- 172.21.15.113:0/334544115 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f108815e110 msgr2=0x7f10880cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f108cb2a700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f108815e110 0x7f10880cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f108cb2a700 1 -- 172.21.15.113:0/334544115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f108806b530 msgr2=0x7f108814d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f108cb2a700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f108806b530 0x7f108814d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f108cb2a700 1 -- 172.21.15.113:0/334544115 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10840094a0 con 0x7f10880afb30 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f108d32b700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f108806b530 0x7f108814d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f108cb2a700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10880afb30 0x7f10881f4260 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f10840090b0 tx=0x7f1084004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f107dffb700 1 -- 172.21.15.113:0/334544115 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10840038a0 con 0x7f10880afb30 2024-01-15T11:25:40.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f107dffb700 1 -- 172.21.15.113:0/334544115 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108401f070 con 0x7f10880afb30 2024-01-15T11:25:40.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f108f58f700 1 -- 172.21.15.113:0/334544115 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108814df40 con 0x7f10880afb30 2024-01-15T11:25:40.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f107dffb700 1 -- 172.21.15.113:0/334544115 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084003c20 con 0x7f10880afb30 2024-01-15T11:25:40.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.474+0000 7f108f58f700 1 -- 172.21.15.113:0/334544115 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108814b620 con 0x7f10880afb30 2024-01-15T11:25:40.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.475+0000 7f108f58f700 1 -- 172.21.15.113:0/334544115 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1088069020 con 0x7f10880afb30 2024-01-15T11:25:40.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.476+0000 7f107dffb700 1 -- 172.21.15.113:0/334544115 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f108401a430 con 0x7f10880afb30 2024-01-15T11:25:40.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.476+0000 7f107dffb700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f106c06b350 0x7f106c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.476+0000 7f107dffb700 1 -- 172.21.15.113:0/334544115 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1084021070 con 0x7f10880afb30 2024-01-15T11:25:40.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.476+0000 7f108d32b700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f106c06b350 0x7f106c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.476+0000 7f108d32b700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f106c06b350 0x7f106c06d810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f108815d490 tx=0x7f1078008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:40.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.481+0000 7f107dffb700 1 -- 172.21.15.113:0/334544115 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10840563a0 con 0x7f10880afb30 2024-01-15T11:25:40.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.669+0000 7f108f58f700 1 -- 172.21.15.113:0/334544115 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10880b2200 con 0x7f10880afb30 2024-01-15T11:25:40.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.669+0000 7f107dffb700 1 -- 172.21.15.113:0/334544115 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f10840253e0 con 0x7f10880afb30 2024-01-15T11:25:40.668 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:40.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.671+0000 7f108f58f700 1 -- 172.21.15.113:0/334544115 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f106c06b350 msgr2=0x7f106c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.671+0000 7f108f58f700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f106c06b350 0x7f106c06d810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f108815d490 tx=0x7f1078008040 comp rx=0 tx=0).stop 2024-01-15T11:25:40.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.671+0000 7f108f58f700 1 -- 172.21.15.113:0/334544115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10880afb30 msgr2=0x7f10881f4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.671+0000 7f108f58f700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10880afb30 0x7f10881f4260 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f10840090b0 tx=0x7f1084004750 comp rx=0 tx=0).stop 2024-01-15T11:25:40.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.671+0000 7f108f58f700 1 -- 172.21.15.113:0/334544115 shutdown_connections 2024-01-15T11:25:40.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.671+0000 7f108f58f700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f106c06b350 0x7f106c06d810 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.671+0000 7f108f58f700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f108815e110 0x7f10880cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.671+0000 7f108f58f700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f108806b530 0x7f108814d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.671+0000 7f108f58f700 1 --2- 172.21.15.113:0/334544115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10880afb30 0x7f10881f4260 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.671+0000 7f108f58f700 1 -- 172.21.15.113:0/334544115 >> 172.21.15.113:0/334544115 conn(0x7f10880b3250 msgr2=0x7f108815d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:40.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.671+0000 7f108f58f700 1 -- 172.21.15.113:0/334544115 shutdown_connections 2024-01-15T11:25:40.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.671+0000 7f108f58f700 1 -- 172.21.15.113:0/334544115 wait complete. 2024-01-15T11:25:40.682 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T11:25:40.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.806+0000 7fc02b868700 1 Processor -- start 2024-01-15T11:25:40.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.807+0000 7fc02b868700 1 -- start start 2024-01-15T11:25:40.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.807+0000 7fc02b868700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc024161690 0x7fc0240d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.807+0000 7fc02b868700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc02409b110 0x7fc0240d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc02b868700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0240d0a70 con 0x7fc024161a70 2024-01-15T11:25:40.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc02b868700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0240d1c10 con 0x7fc024161690 2024-01-15T11:25:40.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc02b868700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240b2ab0 con 0x7fc02409b110 2024-01-15T11:25:40.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc029e05700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc02409b110 0x7fc0240d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc029e05700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc02409b110 0x7fc0240d2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:48188/0 (socket says 172.21.15.113:48188) 2024-01-15T11:25:40.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc029e05700 1 -- 172.21.15.113:0/1185457259 learned_addr learned my addr 172.21.15.113:0/1185457259 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:40.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc029604700 1 --2- 172.21.15.113:0/1185457259 >> v2:172.21.15.141:3300/0 conn(0x7fc024161690 0x7fc0240d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc029e05700 1 -- 172.21.15.113:0/1185457259 >> v2:172.21.15.141:3300/0 conn(0x7fc024161690 msgr2=0x7fc0240d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:40.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc029e05700 1 --2- 172.21.15.113:0/1185457259 >> v2:172.21.15.141:3300/0 conn(0x7fc024161690 0x7fc0240d0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc029e05700 1 -- 172.21.15.113:0/1185457259 >> v1:172.21.15.141:6789/0 conn(0x7fc024161a70 legacy=0x7fc02409aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc029e05700 1 -- 172.21.15.113:0/1185457259 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0240b2d90 con 0x7fc02409b110 2024-01-15T11:25:40.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc029e05700 1 --2- 172.21.15.113:0/1185457259 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc02409b110 0x7fc0240d2970 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc01c009e50 tx=0x7fc01c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3374203b4c107bc7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:40.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc01bfff700 1 -- 172.21.15.113:0/1185457259 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01c00e070 con 0x7fc02409b110 2024-01-15T11:25:40.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.808+0000 7fc01bfff700 1 -- 172.21.15.113:0/1185457259 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01c00b910 con 0x7fc02409b110 2024-01-15T11:25:40.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.809+0000 7fc01bfff700 1 -- 172.21.15.113:0/1185457259 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01c00bcc0 con 0x7fc02409b110 2024-01-15T11:25:40.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.809+0000 7fc02b868700 1 -- 172.21.15.113:0/1185457259 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc02409b110 msgr2=0x7fc0240d2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.809+0000 7fc02b868700 1 --2- 172.21.15.113:0/1185457259 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc02409b110 0x7fc0240d2970 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc01c009e50 tx=0x7fc01c00b330 comp rx=0 tx=0).stop 2024-01-15T11:25:40.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.809+0000 7fc02b868700 1 -- 172.21.15.113:0/1185457259 shutdown_connections 2024-01-15T11:25:40.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.809+0000 7fc02b868700 1 --2- 172.21.15.113:0/1185457259 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc02409b110 0x7fc0240d2970 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.809+0000 7fc02b868700 1 --2- 172.21.15.113:0/1185457259 >> v2:172.21.15.141:3300/0 conn(0x7fc024161690 0x7fc0240d0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.809+0000 7fc02b868700 1 -- 172.21.15.113:0/1185457259 >> 172.21.15.113:0/1185457259 conn(0x7fc0240d1e50 msgr2=0x7fc0240d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:40.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.809+0000 7fc02b868700 1 -- 172.21.15.113:0/1185457259 shutdown_connections 2024-01-15T11:25:40.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.809+0000 7fc02b868700 1 -- 172.21.15.113:0/1185457259 wait complete. 2024-01-15T11:25:40.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.810+0000 7fc02b868700 1 Processor -- start 2024-01-15T11:25:40.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.811+0000 7fc02b868700 1 -- start start 2024-01-15T11:25:40.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.811+0000 7fc02b868700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02409b110 0x7fc0240cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.811+0000 7fc02b868700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc024161690 0x7fc0240c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.811+0000 7fc02b868700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc024161a70 0x7fc0240b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.811+0000 7fc02b868700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240c7b50 con 0x7fc024161690 2024-01-15T11:25:40.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.811+0000 7fc02b868700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240b7ab0 con 0x7fc02409b110 2024-01-15T11:25:40.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.811+0000 7fc02b868700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240c9b30 con 0x7fc024161a70 2024-01-15T11:25:40.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.811+0000 7fc028e03700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc024161690 0x7fc0240c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.811+0000 7fc028e03700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc024161690 0x7fc0240c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59246/0 (socket says 172.21.15.113:59246) 2024-01-15T11:25:40.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.811+0000 7fc028e03700 1 -- 172.21.15.113:0/670587779 learned_addr learned my addr 172.21.15.113:0/670587779 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:40.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc029e05700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc024161a70 0x7fc0240b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc029604700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02409b110 0x7fc0240cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc028e03700 1 -- 172.21.15.113:0/670587779 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc024161a70 msgr2=0x7fc0240b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc028e03700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc024161a70 0x7fc0240b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc028e03700 1 -- 172.21.15.113:0/670587779 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02409b110 msgr2=0x7fc0240cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:40.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc028e03700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02409b110 0x7fc0240cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc028e03700 1 -- 172.21.15.113:0/670587779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c009b30 con 0x7fc024161690 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc028e03700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc024161690 0x7fc0240c7f70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc02000c450 tx=0x7fc02000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc019ffb700 1 -- 172.21.15.113:0/670587779 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0200052d0 con 0x7fc024161690 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc019ffb700 1 -- 172.21.15.113:0/670587779 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc020005430 con 0x7fc024161690 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc02b868700 1 -- 172.21.15.113:0/670587779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0240c71b0 con 0x7fc024161690 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc019ffb700 1 -- 172.21.15.113:0/670587779 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0200038c0 con 0x7fc024161690 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.812+0000 7fc02b868700 1 -- 172.21.15.113:0/670587779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0240c7770 con 0x7fc024161690 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.813+0000 7fc02b868700 1 -- 172.21.15.113:0/670587779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0240c7c90 con 0x7fc024161690 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.814+0000 7fc019ffb700 1 -- 172.21.15.113:0/670587779 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc020003a20 con 0x7fc024161690 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.814+0000 7fc019ffb700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc00806b350 0x7fc00806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.814+0000 7fc019ffb700 1 -- 172.21.15.113:0/670587779 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc020088eb0 con 0x7fc024161690 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.814+0000 7fc029604700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc00806b350 0x7fc00806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:40.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.814+0000 7fc029604700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc00806b350 0x7fc00806d810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc014009be0 tx=0x7fc014009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:40.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:40.818+0000 7fc019ffb700 1 -- 172.21.15.113:0/670587779 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc0240c7c90 con 0x7fc024161690 2024-01-15T11:25:41.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.007+0000 7fc02b868700 1 -- 172.21.15.113:0/670587779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc0240c7c90 con 0x7fc024161690 2024-01-15T11:25:41.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.007+0000 7fc019ffb700 1 -- 172.21.15.113:0/670587779 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc020017020 con 0x7fc024161690 2024-01-15T11:25:41.006 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-15T11:25:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.009+0000 7fc02b868700 1 -- 172.21.15.113:0/670587779 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc00806b350 msgr2=0x7fc00806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.009+0000 7fc02b868700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc00806b350 0x7fc00806d810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc014009be0 tx=0x7fc014009210 comp rx=0 tx=0).stop 2024-01-15T11:25:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.009+0000 7fc02b868700 1 -- 172.21.15.113:0/670587779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc024161690 msgr2=0x7fc0240c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.009+0000 7fc02b868700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc024161690 0x7fc0240c7f70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc02000c450 tx=0x7fc02000c760 comp rx=0 tx=0).stop 2024-01-15T11:25:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.010+0000 7fc02b868700 1 -- 172.21.15.113:0/670587779 shutdown_connections 2024-01-15T11:25:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.010+0000 7fc02b868700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc00806b350 0x7fc00806d810 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.010+0000 7fc02b868700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc024161a70 0x7fc0240b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.010+0000 7fc02b868700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc02409b110 0x7fc0240cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.010+0000 7fc02b868700 1 --2- 172.21.15.113:0/670587779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc024161690 0x7fc0240c7f70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.010+0000 7fc02b868700 1 -- 172.21.15.113:0/670587779 >> 172.21.15.113:0/670587779 conn(0x7fc0240d1e50 msgr2=0x7fc024061af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.010+0000 7fc02b868700 1 -- 172.21.15.113:0/670587779 shutdown_connections 2024-01-15T11:25:41.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.010+0000 7fc02b868700 1 -- 172.21.15.113:0/670587779 wait complete. 2024-01-15T11:25:41.021 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T11:25:41.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.146+0000 7f743f64e700 1 Processor -- start 2024-01-15T11:25:41.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.146+0000 7f743f64e700 1 -- start start 2024-01-15T11:25:41.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f743f64e700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f743809b110 0x7f74380d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f743f64e700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74380d0a70 con 0x7f7438161690 2024-01-15T11:25:41.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f743f64e700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74380d1c10 con 0x7f7438161a70 2024-01-15T11:25:41.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f743f64e700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74380b2ab0 con 0x7f743809b110 2024-01-15T11:25:41.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f743d3ea700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f7438161690 0x7f743809aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42514/0 (socket says 172.21.15.113:42514) 2024-01-15T11:25:41.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f743d3ea700 1 -- 172.21.15.113:0/1194283525 learned_addr learned my addr 172.21.15.113:0/1194283525 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:41.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f743dbeb700 1 --2- 172.21.15.113:0/1194283525 >> v2:172.21.15.141:3300/0 conn(0x7f743809b110 0x7f74380d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f742ffff700 1 -- 172.21.15.113:0/1194283525 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256862061 0 0) 0x7f74380d0a70 con 0x7f7438161690 2024-01-15T11:25:41.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f742ffff700 1 -- 172.21.15.113:0/1194283525 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74240035b0 con 0x7f7438161690 2024-01-15T11:25:41.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f742ffff700 1 -- 172.21.15.113:0/1194283525 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2864238537 0 0) 0x7f74240035b0 con 0x7f7438161690 2024-01-15T11:25:41.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f743dbeb700 1 -- 172.21.15.113:0/1194283525 >> v1:172.21.15.141:6789/0 conn(0x7f7438161a70 legacy=0x7f74380cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f743dbeb700 1 -- 172.21.15.113:0/1194283525 >> v1:172.21.15.113:6789/0 conn(0x7f7438161690 legacy=0x7f743809aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.147+0000 7f743dbeb700 1 -- 172.21.15.113:0/1194283525 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74380b2d90 con 0x7f743809b110 2024-01-15T11:25:41.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.148+0000 7f743dbeb700 1 --2- 172.21.15.113:0/1194283525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743809b110 0x7f74380d0f20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7430009d90 tx=0x7f743000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=139e3ca1a4c480f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:41.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.148+0000 7f742ffff700 1 -- 172.21.15.113:0/1194283525 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743000e070 con 0x7f743809b110 2024-01-15T11:25:41.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.148+0000 7f742ffff700 1 -- 172.21.15.113:0/1194283525 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7430003a20 con 0x7f743809b110 2024-01-15T11:25:41.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.148+0000 7f742ffff700 1 -- 172.21.15.113:0/1194283525 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434001670 con 0x7f743809b110 2024-01-15T11:25:41.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.148+0000 7f743f64e700 1 -- 172.21.15.113:0/1194283525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743809b110 msgr2=0x7f74380d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.148+0000 7f743f64e700 1 --2- 172.21.15.113:0/1194283525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743809b110 0x7f74380d0f20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7430009d90 tx=0x7f743000b330 comp rx=0 tx=0).stop 2024-01-15T11:25:41.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.148+0000 7f743f64e700 1 -- 172.21.15.113:0/1194283525 shutdown_connections 2024-01-15T11:25:41.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.148+0000 7f743f64e700 1 --2- 172.21.15.113:0/1194283525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743809b110 0x7f74380d0f20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.148+0000 7f743f64e700 1 -- 172.21.15.113:0/1194283525 >> 172.21.15.113:0/1194283525 conn(0x7f74380d1e50 msgr2=0x7f74380d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:41.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.149+0000 7f743f64e700 1 -- 172.21.15.113:0/1194283525 shutdown_connections 2024-01-15T11:25:41.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.149+0000 7f743f64e700 1 -- 172.21.15.113:0/1194283525 wait complete. 2024-01-15T11:25:41.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.150+0000 7f743f64e700 1 Processor -- start 2024-01-15T11:25:41.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.150+0000 7f743f64e700 1 -- start start 2024-01-15T11:25:41.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.150+0000 7f743f64e700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743809b110 0x7f74381503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.150+0000 7f743f64e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7438161690 0x7f74380b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.150+0000 7f743f64e700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7438161a70 0x7f74380b6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743f64e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7438150910 con 0x7f7438161690 2024-01-15T11:25:41.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743f64e700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74380b53f0 con 0x7f743809b110 2024-01-15T11:25:41.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743f64e700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74380b56d0 con 0x7f7438161a70 2024-01-15T11:25:41.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743cbe9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7438161690 0x7f74380b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743cbe9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7438161690 0x7f74380b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59262/0 (socket says 172.21.15.113:59262) 2024-01-15T11:25:41.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743cbe9700 1 -- 172.21.15.113:0/2504423536 learned_addr learned my addr 172.21.15.113:0/2504423536 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743d3ea700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743809b110 0x7f74381503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743dbeb700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7438161a70 0x7f74380b6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743cbe9700 1 -- 172.21.15.113:0/2504423536 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7438161a70 msgr2=0x7f74380b6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743cbe9700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7438161a70 0x7f74380b6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743cbe9700 1 -- 172.21.15.113:0/2504423536 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743809b110 msgr2=0x7f74381503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743cbe9700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743809b110 0x7f74381503d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743cbe9700 1 -- 172.21.15.113:0/2504423536 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7430009a70 con 0x7f7438161690 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743cbe9700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7438161690 0x7f74380b4af0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f742800c550 tx=0x7f742800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f742dffb700 1 -- 172.21.15.113:0/2504423536 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74280054e0 con 0x7f7438161690 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f742dffb700 1 -- 172.21.15.113:0/2504423536 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7428003680 con 0x7f7438161690 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f743f64e700 1 -- 172.21.15.113:0/2504423536 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74380b59b0 con 0x7f7438161690 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.151+0000 7f742dffb700 1 -- 172.21.15.113:0/2504423536 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428003a30 con 0x7f7438161690 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.152+0000 7f743f64e700 1 -- 172.21.15.113:0/2504423536 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74380b5f70 con 0x7f7438161690 2024-01-15T11:25:41.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.152+0000 7f743f64e700 1 -- 172.21.15.113:0/2504423536 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74380b6190 con 0x7f7438161690 2024-01-15T11:25:41.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.153+0000 7f742dffb700 1 -- 172.21.15.113:0/2504423536 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7428003b90 con 0x7f7438161690 2024-01-15T11:25:41.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.153+0000 7f742dffb700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f741c06b350 0x7f741c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.153+0000 7f742dffb700 1 -- 172.21.15.113:0/2504423536 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74280891f0 con 0x7f7438161690 2024-01-15T11:25:41.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.153+0000 7f743d3ea700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f741c06b350 0x7f741c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.154+0000 7f743d3ea700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f741c06b350 0x7f741c06d810 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7434001eb0 tx=0x7f7434001c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:41.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.158+0000 7f742dffb700 1 -- 172.21.15.113:0/2504423536 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7428053f30 con 0x7f7438161690 2024-01-15T11:25:41.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.347+0000 7f743f64e700 1 -- 172.21.15.113:0/2504423536 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f74380d16b0 con 0x7f7438161690 2024-01-15T11:25:41.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.347+0000 7f742dffb700 1 -- 172.21.15.113:0/2504423536 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7428017020 con 0x7f7438161690 2024-01-15T11:25:41.346 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-15T11:25:41.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.350+0000 7f743f64e700 1 -- 172.21.15.113:0/2504423536 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f741c06b350 msgr2=0x7f741c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.350+0000 7f743f64e700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f741c06b350 0x7f741c06d810 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7434001eb0 tx=0x7f7434001c80 comp rx=0 tx=0).stop 2024-01-15T11:25:41.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.350+0000 7f743f64e700 1 -- 172.21.15.113:0/2504423536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7438161690 msgr2=0x7f74380b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.350+0000 7f743f64e700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7438161690 0x7f74380b4af0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f742800c550 tx=0x7f742800c910 comp rx=0 tx=0).stop 2024-01-15T11:25:41.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.350+0000 7f743f64e700 1 -- 172.21.15.113:0/2504423536 shutdown_connections 2024-01-15T11:25:41.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.350+0000 7f743f64e700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f741c06b350 0x7f741c06d810 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.350+0000 7f743f64e700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7438161a70 0x7f74380b6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.350+0000 7f743f64e700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f743809b110 0x7f74381503d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.350+0000 7f743f64e700 1 --2- 172.21.15.113:0/2504423536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7438161690 0x7f74380b4af0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.350+0000 7f743f64e700 1 -- 172.21.15.113:0/2504423536 >> 172.21.15.113:0/2504423536 conn(0x7f74380d1e50 msgr2=0x7f743815e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:41.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.350+0000 7f743f64e700 1 -- 172.21.15.113:0/2504423536 shutdown_connections 2024-01-15T11:25:41.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.350+0000 7f743f64e700 1 -- 172.21.15.113:0/2504423536 wait complete. 2024-01-15T11:25:41.361 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:41.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.483+0000 7f6411488700 1 Processor -- start 2024-01-15T11:25:41.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.483+0000 7f6411488700 1 -- start start 2024-01-15T11:25:41.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.484+0000 7f6411488700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f640c15e0f0 0x7f640c0b9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.484+0000 7f6411488700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640c06bc00 con 0x7f640c0cf2a0 2024-01-15T11:25:41.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.484+0000 7f6411488700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640c15e4d0 con 0x7f640c0b2b60 2024-01-15T11:25:41.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.484+0000 7f6411488700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640c06aff0 con 0x7f640c15e0f0 2024-01-15T11:25:41.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.484+0000 7f640b7fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f640c0cf2a0 0x7f640c0d1f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42526/0 (socket says 172.21.15.113:42526) 2024-01-15T11:25:41.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.484+0000 7f640b7fe700 1 -- 172.21.15.113:0/2452476866 learned_addr learned my addr 172.21.15.113:0/2452476866 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:41.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.484+0000 7f640affd700 1 --2- 172.21.15.113:0/2452476866 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f640c15e0f0 0x7f640c0b9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.484+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381800049 0 0) 0x7f640c06bc00 con 0x7f640c0cf2a0 2024-01-15T11:25:41.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.484+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f40035b0 con 0x7f640c0cf2a0 2024-01-15T11:25:41.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.484+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2594999117 0 0) 0x7f63f40035b0 con 0x7f640c0cf2a0 2024-01-15T11:25:41.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.484+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f640c15e0f0 msgr2=0x7f640c0b9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:41.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.484+0000 7f6409ffb700 1 --2- 172.21.15.113:0/2452476866 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f640c15e0f0 0x7f640c0b9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 >> v1:172.21.15.141:6789/0 conn(0x7f640c0b2b60 legacy=0x7f640c0ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c06b2d0 con 0x7f640c0cf2a0 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f64040019f0 con 0x7f640c0cf2a0 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 >> v1:172.21.15.113:6789/0 conn(0x7f640c0cf2a0 legacy=0x7f640c0d1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 --2- 172.21.15.113:0/2452476866 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f640c15e0f0 0x7f63f4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 --2- 172.21.15.113:0/2452476866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63f4003f70 0x7f63f4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 --2- 172.21.15.113:0/2452476866 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63f4004d90 0x7f63f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c06aff0 con 0x7f63f4003f70 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f400ba70 con 0x7f63f4004d90 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f400bc30 con 0x7f640c15e0f0 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f640a7fc700 1 --2- 172.21.15.113:0/2452476866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63f4003f70 0x7f63f4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f640affd700 1 --2- 172.21.15.113:0/2452476866 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f640c15e0f0 0x7f63f4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f640b7fe700 1 --2- 172.21.15.113:0/2452476866 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63f4004d90 0x7f63f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f640a7fc700 1 -- 172.21.15.113:0/2452476866 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f640c15e0f0 msgr2=0x7f63f4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f640a7fc700 1 --2- 172.21.15.113:0/2452476866 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f640c15e0f0 0x7f63f4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f640a7fc700 1 -- 172.21.15.113:0/2452476866 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63f4004d90 msgr2=0x7f63f4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f640a7fc700 1 --2- 172.21.15.113:0/2452476866 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63f4004d90 0x7f63f4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f640a7fc700 1 -- 172.21.15.113:0/2452476866 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f40035b0 con 0x7f63f4003f70 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f640a7fc700 1 --2- 172.21.15.113:0/2452476866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63f4003f70 0x7f63f4004850 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f63fc0018d0 tx=0x7f63f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d456f8d9627ea5ab server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc012620 con 0x7f63f4003f70 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc012780 con 0x7f63f4003f70 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.485+0000 7f6409ffb700 1 -- 172.21.15.113:0/2452476866 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc012b30 con 0x7f63f4003f70 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.486+0000 7f6411488700 1 -- 172.21.15.113:0/2452476866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63f4003f70 msgr2=0x7f63f4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.486+0000 7f6411488700 1 --2- 172.21.15.113:0/2452476866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63f4003f70 0x7f63f4004850 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f63fc0018d0 tx=0x7f63f4005880 comp rx=0 tx=0).stop 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.486+0000 7f640b7fe700 1 -- 172.21.15.113:0/2452476866 reap_dead start 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.486+0000 7f6411488700 1 -- 172.21.15.113:0/2452476866 shutdown_connections 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.486+0000 7f6411488700 1 -- 172.21.15.113:0/2452476866 >> 172.21.15.113:0/2452476866 conn(0x7f640c0b5ce0 msgr2=0x7f640c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.486+0000 7f6411488700 1 -- 172.21.15.113:0/2452476866 shutdown_connections 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.486+0000 7f6411488700 1 -- 172.21.15.113:0/2452476866 wait complete. 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.487+0000 7f6411488700 1 Processor -- start 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.487+0000 7f6411488700 1 -- start start 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f6411488700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63f4003f70 0x7f640c152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f6411488700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f640c15a900 0x7f640c15ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f6411488700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f640c15e8f0 0x7f640c15ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f6411488700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c15b2c0 con 0x7f640c15e8f0 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f6411488700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c0c7990 con 0x7f63f4003f70 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f6411488700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640c15f660 con 0x7f640c15a900 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f640b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f640c15e8f0 0x7f640c15ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f640b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f640c15e8f0 0x7f640c15ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59280/0 (socket says 172.21.15.113:59280) 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f640b7fe700 1 -- 172.21.15.113:0/2621128516 learned_addr learned my addr 172.21.15.113:0/2621128516 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:41.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f640affd700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63f4003f70 0x7f640c152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f640b7fe700 1 -- 172.21.15.113:0/2621128516 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f640c15a900 msgr2=0x7f640c15ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f640b7fe700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f640c15a900 0x7f640c15ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f640b7fe700 1 -- 172.21.15.113:0/2621128516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63f4003f70 msgr2=0x7f640c152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f640b7fe700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63f4003f70 0x7f640c152110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f640b7fe700 1 -- 172.21.15.113:0/2621128516 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c0c7c30 con 0x7f640c15e8f0 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.488+0000 7f640affd700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63f4003f70 0x7f640c152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.489+0000 7f640b7fe700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f640c15e8f0 0x7f640c15ed80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f64040013d0 tx=0x7f640400e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.489+0000 7f63f37fe700 1 -- 172.21.15.113:0/2621128516 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640400ecf0 con 0x7f640c15e8f0 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.489+0000 7f63f37fe700 1 -- 172.21.15.113:0/2621128516 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640400ee90 con 0x7f640c15e8f0 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.489+0000 7f6411488700 1 -- 172.21.15.113:0/2621128516 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640c0c7f10 con 0x7f640c15e8f0 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.489+0000 7f63f37fe700 1 -- 172.21.15.113:0/2621128516 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404013660 con 0x7f640c15e8f0 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.489+0000 7f6411488700 1 -- 172.21.15.113:0/2621128516 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640c15f2c0 con 0x7f640c15e8f0 2024-01-15T11:25:41.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.490+0000 7f6411488700 1 -- 172.21.15.113:0/2621128516 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640c0c8100 con 0x7f640c15e8f0 2024-01-15T11:25:41.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.490+0000 7f63f37fe700 1 -- 172.21.15.113:0/2621128516 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f64040137c0 con 0x7f640c15e8f0 2024-01-15T11:25:41.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.490+0000 7f63f37fe700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f63f806b350 0x7f63f806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.490+0000 7f63f37fe700 1 -- 172.21.15.113:0/2621128516 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f640408a100 con 0x7f640c15e8f0 2024-01-15T11:25:41.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.490+0000 7f640affd700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f63f806b350 0x7f63f806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.491+0000 7f640affd700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f63f806b350 0x7f63f806d810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6400009b50 tx=0x7f6400008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:41.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.495+0000 7f63f37fe700 1 -- 172.21.15.113:0/2621128516 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64040550b0 con 0x7f640c15e8f0 2024-01-15T11:25:41.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.687+0000 7f6411488700 1 -- 172.21.15.113:0/2621128516 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f640c0751e0 con 0x7f640c15e8f0 2024-01-15T11:25:41.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.688+0000 7f63f37fe700 1 -- 172.21.15.113:0/2621128516 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f64040586d0 con 0x7f640c15e8f0 2024-01-15T11:25:41.687 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:41.687 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:41.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.690+0000 7f6411488700 1 -- 172.21.15.113:0/2621128516 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f63f806b350 msgr2=0x7f63f806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.690+0000 7f6411488700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f63f806b350 0x7f63f806d810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6400009b50 tx=0x7f6400008040 comp rx=0 tx=0).stop 2024-01-15T11:25:41.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.690+0000 7f6411488700 1 -- 172.21.15.113:0/2621128516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f640c15e8f0 msgr2=0x7f640c15ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.690+0000 7f6411488700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f640c15e8f0 0x7f640c15ed80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f64040013d0 tx=0x7f640400e440 comp rx=0 tx=0).stop 2024-01-15T11:25:41.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.690+0000 7f6411488700 1 -- 172.21.15.113:0/2621128516 shutdown_connections 2024-01-15T11:25:41.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.690+0000 7f6411488700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f63f806b350 0x7f63f806d810 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.690+0000 7f6411488700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f640c15e8f0 0x7f640c15ed80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.690+0000 7f6411488700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f63f4003f70 0x7f640c152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.690+0000 7f6411488700 1 --2- 172.21.15.113:0/2621128516 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f640c15a900 0x7f640c15ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.690+0000 7f6411488700 1 -- 172.21.15.113:0/2621128516 >> 172.21.15.113:0/2621128516 conn(0x7f640c0b5ce0 msgr2=0x7f640c0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:41.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.690+0000 7f6411488700 1 -- 172.21.15.113:0/2621128516 shutdown_connections 2024-01-15T11:25:41.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.690+0000 7f6411488700 1 -- 172.21.15.113:0/2621128516 wait complete. 2024-01-15T11:25:41.701 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T11:25:13.398220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T11:25:14.589439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:41.701 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:41.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.827+0000 7f72a1a5c700 1 Processor -- start 2024-01-15T11:25:41.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.827+0000 7f72a1a5c700 1 -- start start 2024-01-15T11:25:41.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f72a1a5c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f729c161510 0x7f729c0cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f72a1a5c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f729c0b98e0 0x7f729c0cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f72a1a5c700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c162af0 con 0x7f729c0b59d0 2024-01-15T11:25:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f72a1a5c700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f729c0d13e0 con 0x7f729c0b98e0 2024-01-15T11:25:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f72a1a5c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729c161f70 con 0x7f729c161510 2024-01-15T11:25:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f729a7fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f729c0b98e0 0x7f729c0cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f729a7fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f729c0b98e0 0x7f729c0cf160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59282/0 (socket says 172.21.15.113:59282) 2024-01-15T11:25:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f729a7fc700 1 -- 172.21.15.113:0/1588169644 learned_addr learned my addr 172.21.15.113:0/1588169644 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f729a7fc700 1 -- 172.21.15.113:0/1588169644 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f729c161510 msgr2=0x7f729c0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f729affd700 1 --2- 172.21.15.113:0/1588169644 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f729c161510 0x7f729c0cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f729a7fc700 1 --2- 172.21.15.113:0/1588169644 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f729c161510 0x7f729c0cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f729a7fc700 1 -- 172.21.15.113:0/1588169644 >> v1:172.21.15.141:6789/0 conn(0x7f729c0b59d0 legacy=0x7f729c0b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f729a7fc700 1 -- 172.21.15.113:0/1588169644 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c162250 con 0x7f729c0b98e0 2024-01-15T11:25:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.828+0000 7f729a7fc700 1 --2- 172.21.15.113:0/1588169644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f729c0b98e0 0x7f729c0cf160 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f728c0098f0 tx=0x7f728c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b75b087c966147 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.829+0000 7f7299ffb700 1 -- 172.21.15.113:0/1588169644 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728c00b8a0 con 0x7f729c0b98e0 2024-01-15T11:25:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.829+0000 7f7299ffb700 1 -- 172.21.15.113:0/1588169644 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728c00ba00 con 0x7f729c0b98e0 2024-01-15T11:25:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.829+0000 7f7299ffb700 1 -- 172.21.15.113:0/1588169644 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728c00bd90 con 0x7f729c0b98e0 2024-01-15T11:25:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.829+0000 7f72a1a5c700 1 -- 172.21.15.113:0/1588169644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f729c0b98e0 msgr2=0x7f729c0cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.829+0000 7f72a1a5c700 1 --2- 172.21.15.113:0/1588169644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f729c0b98e0 0x7f729c0cf160 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f728c0098f0 tx=0x7f728c009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.829+0000 7f72a1a5c700 1 -- 172.21.15.113:0/1588169644 shutdown_connections 2024-01-15T11:25:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.829+0000 7f72a1a5c700 1 --2- 172.21.15.113:0/1588169644 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f729c161510 0x7f729c0cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.829+0000 7f72a1a5c700 1 --2- 172.21.15.113:0/1588169644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f729c0b98e0 0x7f729c0cf160 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.829+0000 7f72a1a5c700 1 -- 172.21.15.113:0/1588169644 >> 172.21.15.113:0/1588169644 conn(0x7f729c06b410 msgr2=0x7f729c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.829+0000 7f72a1a5c700 1 -- 172.21.15.113:0/1588169644 shutdown_connections 2024-01-15T11:25:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.829+0000 7f72a1a5c700 1 -- 172.21.15.113:0/1588169644 wait complete. 2024-01-15T11:25:41.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.830+0000 7f72a1a5c700 1 Processor -- start 2024-01-15T11:25:41.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.830+0000 7f72a1a5c700 1 -- start start 2024-01-15T11:25:41.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.831+0000 7f72a1a5c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c0b59d0 0x7f729c0c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.831+0000 7f72a1a5c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f729c0b98e0 0x7f729c158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.831+0000 7f72a1a5c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f729c161510 0x7f729c158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.831+0000 7f72a1a5c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c1543a0 con 0x7f729c161510 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.831+0000 7f72a1a5c700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c0cde40 con 0x7f729c0b59d0 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.831+0000 7f72a1a5c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729c0cdb60 con 0x7f729c0b98e0 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.831+0000 7f729b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f729c161510 0x7f729c158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.831+0000 7f729b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f729c161510 0x7f729c158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59284/0 (socket says 172.21.15.113:59284) 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.831+0000 7f729b7fe700 1 -- 172.21.15.113:0/3708387369 learned_addr learned my addr 172.21.15.113:0/3708387369 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.831+0000 7f729affd700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c0b59d0 0x7f729c0c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.831+0000 7f729a7fc700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f729c0b98e0 0x7f729c158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.832+0000 7f729b7fe700 1 -- 172.21.15.113:0/3708387369 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f729c0b98e0 msgr2=0x7f729c158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.832+0000 7f729b7fe700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f729c0b98e0 0x7f729c158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.832+0000 7f729b7fe700 1 -- 172.21.15.113:0/3708387369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c0b59d0 msgr2=0x7f729c0c6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.832+0000 7f729b7fe700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c0b59d0 0x7f729c0c6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.832+0000 7f729b7fe700 1 -- 172.21.15.113:0/3708387369 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728c009550 con 0x7f729c161510 2024-01-15T11:25:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.832+0000 7f729b7fe700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f729c161510 0x7f729c158990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f729400cc50 tx=0x7f729400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:41.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.832+0000 7f7283fff700 1 -- 172.21.15.113:0/3708387369 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729400ad80 con 0x7f729c161510 2024-01-15T11:25:41.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.832+0000 7f7283fff700 1 -- 172.21.15.113:0/3708387369 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729400ce50 con 0x7f729c161510 2024-01-15T11:25:41.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.832+0000 7f72a1a5c700 1 -- 172.21.15.113:0/3708387369 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f729c153d50 con 0x7f729c161510 2024-01-15T11:25:41.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.832+0000 7f7283fff700 1 -- 172.21.15.113:0/3708387369 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7294003c40 con 0x7f729c161510 2024-01-15T11:25:41.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.832+0000 7f72a1a5c700 1 -- 172.21.15.113:0/3708387369 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f729c0c6ed0 con 0x7f729c161510 2024-01-15T11:25:41.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.833+0000 7f72a1a5c700 1 -- 172.21.15.113:0/3708387369 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729c068450 con 0x7f729c161510 2024-01-15T11:25:41.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.833+0000 7f7283fff700 1 -- 172.21.15.113:0/3708387369 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f72940050d0 con 0x7f729c161510 2024-01-15T11:25:41.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.834+0000 7f7283fff700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f728806b350 0x7f728806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:41.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.834+0000 7f7283fff700 1 -- 172.21.15.113:0/3708387369 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7294089c20 con 0x7f729c161510 2024-01-15T11:25:41.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.834+0000 7f729affd700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f728806b350 0x7f728806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:41.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.834+0000 7f729affd700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f728806b350 0x7f728806d810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f729000a950 tx=0x7f7290008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:41.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:41.838+0000 7f7283fff700 1 -- 172.21.15.113:0/3708387369 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7294054b20 con 0x7f729c161510 2024-01-15T11:25:42.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.026+0000 7f72a1a5c700 1 -- 172.21.15.113:0/3708387369 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f729c069020 con 0x7f729c161510 2024-01-15T11:25:42.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.026+0000 7f7283fff700 1 -- 172.21.15.113:0/3708387369 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7294058140 con 0x7f729c161510 2024-01-15T11:25:42.025 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:42.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.028+0000 7f72a1a5c700 1 -- 172.21.15.113:0/3708387369 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f728806b350 msgr2=0x7f728806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.028+0000 7f72a1a5c700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f728806b350 0x7f728806d810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f729000a950 tx=0x7f7290008040 comp rx=0 tx=0).stop 2024-01-15T11:25:42.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.028+0000 7f72a1a5c700 1 -- 172.21.15.113:0/3708387369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f729c161510 msgr2=0x7f729c158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.028+0000 7f72a1a5c700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f729c161510 0x7f729c158990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f729400cc50 tx=0x7f729400a340 comp rx=0 tx=0).stop 2024-01-15T11:25:42.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.028+0000 7f72a1a5c700 1 -- 172.21.15.113:0/3708387369 shutdown_connections 2024-01-15T11:25:42.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.028+0000 7f72a1a5c700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f728806b350 0x7f728806d810 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.028+0000 7f72a1a5c700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f729c161510 0x7f729c158990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.028+0000 7f72a1a5c700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f729c0b59d0 0x7f729c0c6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.028+0000 7f72a1a5c700 1 --2- 172.21.15.113:0/3708387369 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f729c0b98e0 0x7f729c158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.028+0000 7f72a1a5c700 1 -- 172.21.15.113:0/3708387369 >> 172.21.15.113:0/3708387369 conn(0x7f729c06b410 msgr2=0x7f729c0b5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:42.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.029+0000 7f72a1a5c700 1 -- 172.21.15.113:0/3708387369 shutdown_connections 2024-01-15T11:25:42.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.029+0000 7f72a1a5c700 1 -- 172.21.15.113:0/3708387369 wait complete. 2024-01-15T11:25:42.039 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T11:25:42.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.167+0000 7f6829776700 1 Processor -- start 2024-01-15T11:25:42.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.167+0000 7f6829776700 1 -- start start 2024-01-15T11:25:42.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f6829776700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6824161510 0x7f68240cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f6829776700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68240b98e0 0x7f68240cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f6829776700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f68240b59d0 0x7f68240b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f6829776700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6824162af0 con 0x7f68240b59d0 2024-01-15T11:25:42.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f6829776700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68240d13e0 con 0x7f6824161510 2024-01-15T11:25:42.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f6829776700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6824161f70 con 0x7f68240b98e0 2024-01-15T11:25:42.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f68237fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f68240b59d0 0x7f68240b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f68237fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f68240b59d0 0x7f68240b5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59286/0 (socket says 172.21.15.113:59286) 2024-01-15T11:25:42.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f68237fe700 1 -- 172.21.15.113:0/551297539 learned_addr learned my addr 172.21.15.113:0/551297539 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:42.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f6822ffd700 1 --2- 172.21.15.113:0/551297539 >> v2:172.21.15.141:3300/0 conn(0x7f6824161510 0x7f68240cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f68227fc700 1 --2- 172.21.15.113:0/551297539 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68240b98e0 0x7f68240cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f68237fe700 1 -- 172.21.15.113:0/551297539 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68240b98e0 msgr2=0x7f68240cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:42.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f68237fe700 1 --2- 172.21.15.113:0/551297539 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68240b98e0 0x7f68240cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f68237fe700 1 -- 172.21.15.113:0/551297539 >> v2:172.21.15.141:3300/0 conn(0x7f6824161510 msgr2=0x7f68240cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f68237fe700 1 --2- 172.21.15.113:0/551297539 >> v2:172.21.15.141:3300/0 conn(0x7f6824161510 0x7f68240cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f68237fe700 1 -- 172.21.15.113:0/551297539 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6824162250 con 0x7f68240b59d0 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f68237fe700 1 --2- 172.21.15.113:0/551297539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68240b59d0 0x7f68240b5e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f68240d49f0 tx=0x7f681c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71845e658cec35 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f6821ffb700 1 -- 172.21.15.113:0/551297539 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c00e070 con 0x7f68240b59d0 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f6821ffb700 1 -- 172.21.15.113:0/551297539 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681c00b9c0 con 0x7f68240b59d0 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.168+0000 7f6821ffb700 1 -- 172.21.15.113:0/551297539 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c00bd70 con 0x7f68240b59d0 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.169+0000 7f6829776700 1 -- 172.21.15.113:0/551297539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68240b59d0 msgr2=0x7f68240b5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.169+0000 7f6829776700 1 --2- 172.21.15.113:0/551297539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68240b59d0 0x7f68240b5e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f68240d49f0 tx=0x7f681c00b330 comp rx=0 tx=0).stop 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.169+0000 7f6829776700 1 -- 172.21.15.113:0/551297539 shutdown_connections 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.169+0000 7f6829776700 1 --2- 172.21.15.113:0/551297539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68240b59d0 0x7f68240b5e60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.169+0000 7f6829776700 1 --2- 172.21.15.113:0/551297539 >> v2:172.21.15.141:3300/0 conn(0x7f6824161510 0x7f68240cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.169+0000 7f6829776700 1 --2- 172.21.15.113:0/551297539 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68240b98e0 0x7f68240cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.169+0000 7f6829776700 1 -- 172.21.15.113:0/551297539 >> 172.21.15.113:0/551297539 conn(0x7f682406b410 msgr2=0x7f682406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:42.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.169+0000 7f6829776700 1 -- 172.21.15.113:0/551297539 shutdown_connections 2024-01-15T11:25:42.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.169+0000 7f6829776700 1 -- 172.21.15.113:0/551297539 wait complete. 2024-01-15T11:25:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.171+0000 7f6829776700 1 Processor -- start 2024-01-15T11:25:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.171+0000 7f6829776700 1 -- start start 2024-01-15T11:25:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f6829776700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68240b59d0 0x7f68240acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f6829776700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68240b98e0 0x7f68241f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f6829776700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6824161510 0x7f6824151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f6829776700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682415db30 con 0x7f6824161510 2024-01-15T11:25:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f6829776700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682415dd80 con 0x7f68240b59d0 2024-01-15T11:25:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f6829776700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682415e060 con 0x7f68240b98e0 2024-01-15T11:25:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f68237fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6824161510 0x7f6824151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f68237fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6824161510 0x7f6824151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59292/0 (socket says 172.21.15.113:59292) 2024-01-15T11:25:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f68237fe700 1 -- 172.21.15.113:0/102632473 learned_addr learned my addr 172.21.15.113:0/102632473 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f68227fc700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68240b98e0 0x7f68241f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f6822ffd700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68240b59d0 0x7f68240acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f68237fe700 1 -- 172.21.15.113:0/102632473 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68240b98e0 msgr2=0x7f68241f3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f68237fe700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68240b98e0 0x7f68241f3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f68237fe700 1 -- 172.21.15.113:0/102632473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68240b59d0 msgr2=0x7f68240acaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f68237fe700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68240b59d0 0x7f68240acaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f68237fe700 1 -- 172.21.15.113:0/102632473 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c009b30 con 0x7f6824161510 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f6822ffd700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68240b59d0 0x7f68240acaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f68237fe700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6824161510 0x7f6824151f80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f68240d17d0 tx=0x7f681c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f680b7fe700 1 -- 172.21.15.113:0/102632473 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c0099b0 con 0x7f6824161510 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f680b7fe700 1 -- 172.21.15.113:0/102632473 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681c004250 con 0x7f6824161510 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f6829776700 1 -- 172.21.15.113:0/102632473 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f682415e340 con 0x7f6824161510 2024-01-15T11:25:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f680b7fe700 1 -- 172.21.15.113:0/102632473 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c004600 con 0x7f6824161510 2024-01-15T11:25:42.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.172+0000 7f6829776700 1 -- 172.21.15.113:0/102632473 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68240ad690 con 0x7f6824161510 2024-01-15T11:25:42.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.173+0000 7f6829776700 1 -- 172.21.15.113:0/102632473 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6824068450 con 0x7f6824161510 2024-01-15T11:25:42.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.174+0000 7f680b7fe700 1 -- 172.21.15.113:0/102632473 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f681c004760 con 0x7f6824161510 2024-01-15T11:25:42.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.174+0000 7f680b7fe700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f681006b350 0x7f681006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.174+0000 7f680b7fe700 1 -- 172.21.15.113:0/102632473 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f681c08b000 con 0x7f6824161510 2024-01-15T11:25:42.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.174+0000 7f6822ffd700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f681006b350 0x7f681006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.175+0000 7f6822ffd700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f681006b350 0x7f681006d810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6818007a50 tx=0x7f6818008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:42.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.179+0000 7f680b7fe700 1 -- 172.21.15.113:0/102632473 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f681c055f00 con 0x7f6824161510 2024-01-15T11:25:42.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.370+0000 7f6829776700 1 -- 172.21.15.113:0/102632473 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6824069020 con 0x7f6824161510 2024-01-15T11:25:42.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.370+0000 7f680b7fe700 1 -- 172.21.15.113:0/102632473 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f681c020020 con 0x7f6824161510 2024-01-15T11:25:42.369 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-15T11:25:42.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.372+0000 7f6829776700 1 -- 172.21.15.113:0/102632473 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f681006b350 msgr2=0x7f681006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.372+0000 7f6829776700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f681006b350 0x7f681006d810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6818007a50 tx=0x7f6818008040 comp rx=0 tx=0).stop 2024-01-15T11:25:42.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.372+0000 7f6829776700 1 -- 172.21.15.113:0/102632473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6824161510 msgr2=0x7f6824151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.372+0000 7f6829776700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6824161510 0x7f6824151f80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f68240d17d0 tx=0x7f681c004ad0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.372+0000 7f6829776700 1 -- 172.21.15.113:0/102632473 shutdown_connections 2024-01-15T11:25:42.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.372+0000 7f6829776700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f681006b350 0x7f681006d810 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.372+0000 7f6829776700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6824161510 0x7f6824151f80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.372+0000 7f6829776700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f68240b59d0 0x7f68240acaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.372+0000 7f6829776700 1 --2- 172.21.15.113:0/102632473 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f68240b98e0 0x7f68241f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.372+0000 7f6829776700 1 -- 172.21.15.113:0/102632473 >> 172.21.15.113:0/102632473 conn(0x7f682406b410 msgr2=0x7f68240b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:42.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.373+0000 7f6829776700 1 -- 172.21.15.113:0/102632473 shutdown_connections 2024-01-15T11:25:42.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.373+0000 7f6829776700 1 -- 172.21.15.113:0/102632473 wait complete. 2024-01-15T11:25:42.384 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T11:25:42.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.509+0000 7f86ceac3700 1 Processor -- start 2024-01-15T11:25:42.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.509+0000 7f86ceac3700 1 -- start start 2024-01-15T11:25:42.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86ceac3700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c80b5220 0x7f86c80b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86ceac3700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f86c8073f90 0x7f86c806e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86ceac3700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c80b42a0 con 0x7f86c815e280 2024-01-15T11:25:42.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86ceac3700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c80af900 con 0x7f86c8073f90 2024-01-15T11:25:42.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86ceac3700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c8074d30 con 0x7f86c80b5220 2024-01-15T11:25:42.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86cc85f700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f86c815e280 0x7f86c80b4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42530/0 (socket says 172.21.15.113:42530) 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86cc85f700 1 -- 172.21.15.113:0/2786811562 learned_addr learned my addr 172.21.15.113:0/2786811562 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488775429 0 0) 0x7f86bc001110 con 0x7f86c815e280 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b80035b0 con 0x7f86c815e280 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c7fff700 1 --2- 172.21.15.113:0/2786811562 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c80b5220 0x7f86c80b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86cd060700 1 --2- 172.21.15.113:0/2786811562 >> v2:172.21.15.141:3300/0 conn(0x7f86c8073f90 0x7f86c806e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4215536219 0 0) 0x7f86b80035b0 con 0x7f86c815e280 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c80b5220 msgr2=0x7f86c80b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c77fe700 1 --2- 172.21.15.113:0/2786811562 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c80b5220 0x7f86c80b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 >> v2:172.21.15.141:3300/0 conn(0x7f86c8073f90 msgr2=0x7f86c806e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c77fe700 1 --2- 172.21.15.113:0/2786811562 >> v2:172.21.15.141:3300/0 conn(0x7f86c8073f90 0x7f86c806e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c8074490 con 0x7f86c815e280 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f86bc001800 con 0x7f86c815e280 2024-01-15T11:25:42.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 >> v1:172.21.15.113:6789/0 conn(0x7f86c815e280 legacy=0x7f86c80b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c77fe700 1 --2- 172.21.15.113:0/2786811562 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c80b5220 0x7f86b8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.510+0000 7f86c77fe700 1 --2- 172.21.15.113:0/2786811562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86b8003f80 0x7f86b8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c77fe700 1 --2- 172.21.15.113:0/2786811562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b8004d90 0x7f86b8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c80af900 con 0x7f86b8003f80 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c8074d30 con 0x7f86b8004d90 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b800ba10 con 0x7f86c80b5220 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c7fff700 1 --2- 172.21.15.113:0/2786811562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86b8003f80 0x7f86b8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86cc85f700 1 --2- 172.21.15.113:0/2786811562 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c80b5220 0x7f86b8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86cd060700 1 --2- 172.21.15.113:0/2786811562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b8004d90 0x7f86b8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c7fff700 1 -- 172.21.15.113:0/2786811562 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c80b5220 msgr2=0x7f86b8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c7fff700 1 --2- 172.21.15.113:0/2786811562 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c80b5220 0x7f86b8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c7fff700 1 -- 172.21.15.113:0/2786811562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b8004d90 msgr2=0x7f86b8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c7fff700 1 --2- 172.21.15.113:0/2786811562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b8004d90 0x7f86b8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c7fff700 1 -- 172.21.15.113:0/2786811562 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b80035b0 con 0x7f86b8003f80 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c7fff700 1 --2- 172.21.15.113:0/2786811562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86b8003f80 0x7f86b8004850 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f86b4009b60 tx=0x7f86b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd45ccb66df2662f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b400e660 con 0x7f86b8003f80 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b400e7c0 con 0x7f86b8003f80 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.511+0000 7f86c77fe700 1 -- 172.21.15.113:0/2786811562 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b400eb30 con 0x7f86b8003f80 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.512+0000 7f86ceac3700 1 -- 172.21.15.113:0/2786811562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86b8003f80 msgr2=0x7f86b8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.512+0000 7f86ceac3700 1 --2- 172.21.15.113:0/2786811562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86b8003f80 0x7f86b8004850 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f86b4009b60 tx=0x7f86b8005880 comp rx=0 tx=0).stop 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.512+0000 7f86cd060700 1 -- 172.21.15.113:0/2786811562 reap_dead start 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.512+0000 7f86ceac3700 1 -- 172.21.15.113:0/2786811562 shutdown_connections 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.512+0000 7f86ceac3700 1 -- 172.21.15.113:0/2786811562 >> 172.21.15.113:0/2786811562 conn(0x7f86c8072b20 msgr2=0x7f86c806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.512+0000 7f86ceac3700 1 -- 172.21.15.113:0/2786811562 shutdown_connections 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.512+0000 7f86ceac3700 1 -- 172.21.15.113:0/2786811562 wait complete. 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.513+0000 7f86ceac3700 1 Processor -- start 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.513+0000 7f86ceac3700 1 -- start start 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86ceac3700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b8003f80 0x7f86c815ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86ceac3700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c815efd0 0x7f86c8147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86ceac3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c81477a0 0x7f86c8147c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86ceac3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c8148160 con 0x7f86c81477a0 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86ceac3700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c81603d0 con 0x7f86b8003f80 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86ceac3700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c815f7c0 con 0x7f86c815efd0 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86cd060700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c81477a0 0x7f86c8147c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86cd060700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c81477a0 0x7f86c8147c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59300/0 (socket says 172.21.15.113:59300) 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86cd060700 1 -- 172.21.15.113:0/1386846229 learned_addr learned my addr 172.21.15.113:0/1386846229 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86c7fff700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c815efd0 0x7f86c8147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86cc85f700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b8003f80 0x7f86c815ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86cd060700 1 -- 172.21.15.113:0/1386846229 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c815efd0 msgr2=0x7f86c8147260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86cd060700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c815efd0 0x7f86c8147260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86cd060700 1 -- 172.21.15.113:0/1386846229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b8003f80 msgr2=0x7f86c815ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86cd060700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b8003f80 0x7f86c815ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86cd060700 1 -- 172.21.15.113:0/1386846229 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c815faa0 con 0x7f86c81477a0 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86cc85f700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b8003f80 0x7f86c815ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.514+0000 7f86cd060700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c81477a0 0x7f86c8147c20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f86c80743c0 tx=0x7f86c00114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.515+0000 7f86c57fa700 1 -- 172.21.15.113:0/1386846229 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c000e070 con 0x7f86c81477a0 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.515+0000 7f86c57fa700 1 -- 172.21.15.113:0/1386846229 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c0003c30 con 0x7f86c81477a0 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.515+0000 7f86c57fa700 1 -- 172.21.15.113:0/1386846229 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0003e20 con 0x7f86c81477a0 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.515+0000 7f86ceac3700 1 -- 172.21.15.113:0/1386846229 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c815fd80 con 0x7f86c81477a0 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.515+0000 7f86ceac3700 1 -- 172.21.15.113:0/1386846229 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c815f450 con 0x7f86c81477a0 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.515+0000 7f86ceac3700 1 -- 172.21.15.113:0/1386846229 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c8145230 con 0x7f86c81477a0 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.516+0000 7f86c57fa700 1 -- 172.21.15.113:0/1386846229 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f86c0011a40 con 0x7f86c81477a0 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.516+0000 7f86c57fa700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f86ac06b350 0x7f86ac06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.516+0000 7f86c57fa700 1 -- 172.21.15.113:0/1386846229 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86c008a000 con 0x7f86c81477a0 2024-01-15T11:25:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.516+0000 7f86cc85f700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f86ac06b350 0x7f86ac06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.517+0000 7f86cc85f700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f86ac06b350 0x7f86ac06d810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f86bc000fb0 tx=0x7f86bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:42.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.521+0000 7f86c57fa700 1 -- 172.21.15.113:0/1386846229 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86c8145230 con 0x7f86c81477a0 2024-01-15T11:25:42.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.712+0000 7f86ceac3700 1 -- 172.21.15.113:0/1386846229 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f86c8145230 con 0x7f86c81477a0 2024-01-15T11:25:42.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.713+0000 7f86c57fa700 1 -- 172.21.15.113:0/1386846229 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f86c8145230 con 0x7f86c81477a0 2024-01-15T11:25:42.712 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-15T11:25:42.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.715+0000 7f86ceac3700 1 -- 172.21.15.113:0/1386846229 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f86ac06b350 msgr2=0x7f86ac06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.715+0000 7f86ceac3700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f86ac06b350 0x7f86ac06d810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f86bc000fb0 tx=0x7f86bc009040 comp rx=0 tx=0).stop 2024-01-15T11:25:42.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.715+0000 7f86ceac3700 1 -- 172.21.15.113:0/1386846229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c81477a0 msgr2=0x7f86c8147c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.715+0000 7f86ceac3700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c81477a0 0x7f86c8147c20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f86c80743c0 tx=0x7f86c00114b0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.715+0000 7f86ceac3700 1 -- 172.21.15.113:0/1386846229 shutdown_connections 2024-01-15T11:25:42.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.715+0000 7f86ceac3700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f86ac06b350 0x7f86ac06d810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.715+0000 7f86ceac3700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c81477a0 0x7f86c8147c20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.715+0000 7f86ceac3700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f86b8003f80 0x7f86c815ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.715+0000 7f86ceac3700 1 --2- 172.21.15.113:0/1386846229 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f86c815efd0 0x7f86c8147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.715+0000 7f86ceac3700 1 -- 172.21.15.113:0/1386846229 >> 172.21.15.113:0/1386846229 conn(0x7f86c8072b20 msgr2=0x7f86c815d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:42.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.715+0000 7f86ceac3700 1 -- 172.21.15.113:0/1386846229 shutdown_connections 2024-01-15T11:25:42.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.715+0000 7f86ceac3700 1 -- 172.21.15.113:0/1386846229 wait complete. 2024-01-15T11:25:42.726 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:42.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.848+0000 7f37ced43700 1 Processor -- start 2024-01-15T11:25:42.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.848+0000 7f37ced43700 1 -- start start 2024-01-15T11:25:42.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.849+0000 7f37ced43700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f37c8158770 0x7f37c806e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.849+0000 7f37ced43700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f37c8071bd0 0x7f37c806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.849+0000 7f37ced43700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c80b3000 con 0x7f37c815b140 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.849+0000 7f37ced43700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c80b32e0 con 0x7f37c8071bd0 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.849+0000 7f37ced43700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c80b35c0 con 0x7f37c8158770 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.849+0000 7f37cd2e0700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f37c8071bd0 0x7f37c806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.849+0000 7f37cd2e0700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f37c8071bd0 0x7f37c806e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59314/0 (socket says 172.21.15.113:59314) 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.849+0000 7f37cd2e0700 1 -- 172.21.15.113:0/3580335917 learned_addr learned my addr 172.21.15.113:0/3580335917 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.849+0000 7f37ccadf700 1 --2- 172.21.15.113:0/3580335917 >> v2:172.21.15.141:3300/0 conn(0x7f37c8158770 0x7f37c806e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37cd2e0700 1 -- 172.21.15.113:0/3580335917 >> v2:172.21.15.141:3300/0 conn(0x7f37c8158770 msgr2=0x7f37c806e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37cd2e0700 1 --2- 172.21.15.113:0/3580335917 >> v2:172.21.15.141:3300/0 conn(0x7f37c8158770 0x7f37c806e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37cd2e0700 1 -- 172.21.15.113:0/3580335917 >> v1:172.21.15.141:6789/0 conn(0x7f37c815b140 legacy=0x7f37c815e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37cd2e0700 1 -- 172.21.15.113:0/3580335917 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c80b38a0 con 0x7f37c8071bd0 2024-01-15T11:25:42.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37cd2e0700 1 --2- 172.21.15.113:0/3580335917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37c8071bd0 0x7f37c806e9a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f37c80b5c80 tx=0x7f37c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf03df5d326e3aa7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:42.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37c77fe700 1 -- 172.21.15.113:0/3580335917 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c000e070 con 0x7f37c8071bd0 2024-01-15T11:25:42.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37c77fe700 1 -- 172.21.15.113:0/3580335917 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c0003a20 con 0x7f37c8071bd0 2024-01-15T11:25:42.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37c77fe700 1 -- 172.21.15.113:0/3580335917 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c0003dd0 con 0x7f37c8071bd0 2024-01-15T11:25:42.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37ced43700 1 -- 172.21.15.113:0/3580335917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37c8071bd0 msgr2=0x7f37c806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37ced43700 1 --2- 172.21.15.113:0/3580335917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37c8071bd0 0x7f37c806e9a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f37c80b5c80 tx=0x7f37c000b330 comp rx=0 tx=0).stop 2024-01-15T11:25:42.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37ced43700 1 -- 172.21.15.113:0/3580335917 shutdown_connections 2024-01-15T11:25:42.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37ced43700 1 --2- 172.21.15.113:0/3580335917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37c8071bd0 0x7f37c806e9a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37ced43700 1 --2- 172.21.15.113:0/3580335917 >> v2:172.21.15.141:3300/0 conn(0x7f37c8158770 0x7f37c806e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37ced43700 1 -- 172.21.15.113:0/3580335917 >> 172.21.15.113:0/3580335917 conn(0x7f37c80b1740 msgr2=0x7f37c80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:42.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.850+0000 7f37ced43700 1 -- 172.21.15.113:0/3580335917 shutdown_connections 2024-01-15T11:25:42.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.851+0000 7f37ced43700 1 -- 172.21.15.113:0/3580335917 wait complete. 2024-01-15T11:25:42.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.852+0000 7f37ced43700 1 Processor -- start 2024-01-15T11:25:42.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.852+0000 7f37ced43700 1 -- start start 2024-01-15T11:25:42.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.853+0000 7f37ced43700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f37c8071bd0 0x7f37c814e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.853+0000 7f37ced43700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37c8158770 0x7f37c814f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.853+0000 7f37ced43700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37c815b140 0x7f37c815a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.853+0000 7f37ced43700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c814e7c0 con 0x7f37c8158770 2024-01-15T11:25:42.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.853+0000 7f37ced43700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c814eaa0 con 0x7f37c815b140 2024-01-15T11:25:42.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.853+0000 7f37ced43700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c814ed80 con 0x7f37c8071bd0 2024-01-15T11:25:42.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.853+0000 7f37c7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37c8158770 0x7f37c814f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.853+0000 7f37c7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37c8158770 0x7f37c814f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59322/0 (socket says 172.21.15.113:59322) 2024-01-15T11:25:42.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.853+0000 7f37c7fff700 1 -- 172.21.15.113:0/233675125 learned_addr learned my addr 172.21.15.113:0/233675125 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:42.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37cd2e0700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37c815b140 0x7f37c815a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37ccadf700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f37c8071bd0 0x7f37c814e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37c7fff700 1 -- 172.21.15.113:0/233675125 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f37c8071bd0 msgr2=0x7f37c814e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37c7fff700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f37c8071bd0 0x7f37c814e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37c7fff700 1 -- 172.21.15.113:0/233675125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37c815b140 msgr2=0x7f37c815a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37c7fff700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37c815b140 0x7f37c815a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37c7fff700 1 -- 172.21.15.113:0/233675125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c0009b30 con 0x7f37c8158770 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37c7fff700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37c8158770 0x7f37c814f860 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f37b400b4a0 tx=0x7f37b400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37c57fa700 1 -- 172.21.15.113:0/233675125 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4011830 con 0x7f37c8158770 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37c57fa700 1 -- 172.21.15.113:0/233675125 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b4011990 con 0x7f37c8158770 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37c57fa700 1 -- 172.21.15.113:0/233675125 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4011d40 con 0x7f37c8158770 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37ced43700 1 -- 172.21.15.113:0/233675125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c814f060 con 0x7f37c8158770 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.854+0000 7f37ced43700 1 -- 172.21.15.113:0/233675125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c813e440 con 0x7f37c8158770 2024-01-15T11:25:42.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.855+0000 7f37ced43700 1 -- 172.21.15.113:0/233675125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c814f580 con 0x7f37c8158770 2024-01-15T11:25:42.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.855+0000 7f37c57fa700 1 -- 172.21.15.113:0/233675125 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f37b401a400 con 0x7f37c8158770 2024-01-15T11:25:42.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.856+0000 7f37c57fa700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f37ac06b350 0x7f37ac06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:42.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.856+0000 7f37c57fa700 1 -- 172.21.15.113:0/233675125 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37b40899d0 con 0x7f37c8158770 2024-01-15T11:25:42.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.856+0000 7f37ccadf700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f37ac06b350 0x7f37ac06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:42.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.856+0000 7f37ccadf700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f37ac06b350 0x7f37ac06d810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009950 tx=0x7f37bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:42.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:42.860+0000 7f37c57fa700 1 -- 172.21.15.113:0/233675125 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37c814f580 con 0x7f37c8158770 2024-01-15T11:25:43.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.053+0000 7f37ced43700 1 -- 172.21.15.113:0/233675125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37c814f580 con 0x7f37c8158770 2024-01-15T11:25:43.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.054+0000 7f37c57fa700 1 -- 172.21.15.113:0/233675125 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f37c814f580 con 0x7f37c8158770 2024-01-15T11:25:43.053 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:43.053 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:43.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.056+0000 7f37ced43700 1 -- 172.21.15.113:0/233675125 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f37ac06b350 msgr2=0x7f37ac06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.056+0000 7f37ced43700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f37ac06b350 0x7f37ac06d810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009950 tx=0x7f37bc008040 comp rx=0 tx=0).stop 2024-01-15T11:25:43.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.056+0000 7f37ced43700 1 -- 172.21.15.113:0/233675125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37c8158770 msgr2=0x7f37c814f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.056+0000 7f37ced43700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37c8158770 0x7f37c814f860 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f37b400b4a0 tx=0x7f37b400b860 comp rx=0 tx=0).stop 2024-01-15T11:25:43.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.056+0000 7f37ced43700 1 -- 172.21.15.113:0/233675125 shutdown_connections 2024-01-15T11:25:43.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.056+0000 7f37ced43700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f37ac06b350 0x7f37ac06d810 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.056+0000 7f37ced43700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f37c815b140 0x7f37c815a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.056+0000 7f37ced43700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f37c8071bd0 0x7f37c814e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.056+0000 7f37ced43700 1 --2- 172.21.15.113:0/233675125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37c8158770 0x7f37c814f860 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.056+0000 7f37ced43700 1 -- 172.21.15.113:0/233675125 >> 172.21.15.113:0/233675125 conn(0x7f37c80b1740 msgr2=0x7f37c80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:43.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.057+0000 7f37ced43700 1 -- 172.21.15.113:0/233675125 shutdown_connections 2024-01-15T11:25:43.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.057+0000 7f37ced43700 1 -- 172.21.15.113:0/233675125 wait complete. 2024-01-15T11:25:43.068 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T11:25:13.398220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T11:25:14.589439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:43.068 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:43.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.194+0000 7f0505878700 1 Processor -- start 2024-01-15T11:25:43.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.194+0000 7f0505878700 1 -- start start 2024-01-15T11:25:43.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f0505878700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f05000d2100 0x7f05000ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f0505878700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050015e930 con 0x7f050015ded0 2024-01-15T11:25:43.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f0505878700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050015ec10 con 0x7f050015e2b0 2024-01-15T11:25:43.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f0505878700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f050015e710 con 0x7f05000d2100 2024-01-15T11:25:43.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f04ff7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f05000d2100 0x7f05000ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f04feffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f050015ded0 0x7f050006edf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42532/0 (socket says 172.21.15.113:42532) 2024-01-15T11:25:43.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f04feffd700 1 -- 172.21.15.113:0/4248166753 learned_addr learned my addr 172.21.15.113:0/4248166753 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:43.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f04ff7fe700 1 -- 172.21.15.113:0/4248166753 >> v1:172.21.15.141:6789/0 conn(0x7f050015e2b0 legacy=0x7f05000d19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f04ff7fe700 1 -- 172.21.15.113:0/4248166753 >> v1:172.21.15.113:6789/0 conn(0x7f050015ded0 legacy=0x7f050006edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f04ff7fe700 1 -- 172.21.15.113:0/4248166753 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050015eef0 con 0x7f05000d2100 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f04fdffb700 1 -- 172.21.15.113:0/4248166753 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203333952 0 0) 0x7f050015e930 con 0x7f050015ded0 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f04ff7fe700 1 --2- 172.21.15.113:0/4248166753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000d2100 0x7f05000ce790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009d90 tx=0x7f04f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34af960a948977e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.195+0000 7f04fdffb700 1 -- 172.21.15.113:0/4248166753 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f800e070 con 0x7f05000d2100 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.196+0000 7f04fdffb700 1 -- 172.21.15.113:0/4248166753 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f800bb90 con 0x7f05000d2100 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.196+0000 7f04fdffb700 1 -- 172.21.15.113:0/4248166753 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f8003a20 con 0x7f05000d2100 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.196+0000 7f0505878700 1 -- 172.21.15.113:0/4248166753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000d2100 msgr2=0x7f05000ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.196+0000 7f0505878700 1 --2- 172.21.15.113:0/4248166753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000d2100 0x7f05000ce790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009d90 tx=0x7f04f800b330 comp rx=0 tx=0).stop 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.196+0000 7f0505878700 1 -- 172.21.15.113:0/4248166753 shutdown_connections 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.196+0000 7f0505878700 1 --2- 172.21.15.113:0/4248166753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000d2100 0x7f05000ce790 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.196+0000 7f0505878700 1 -- 172.21.15.113:0/4248166753 >> 172.21.15.113:0/4248166753 conn(0x7f050009aa00 msgr2=0x7f050009ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:43.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.196+0000 7f0505878700 1 -- 172.21.15.113:0/4248166753 shutdown_connections 2024-01-15T11:25:43.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.196+0000 7f0505878700 1 -- 172.21.15.113:0/4248166753 wait complete. 2024-01-15T11:25:43.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.197+0000 7f0505878700 1 Processor -- start 2024-01-15T11:25:43.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.197+0000 7f0505878700 1 -- start start 2024-01-15T11:25:43.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.198+0000 7f0505878700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05000d2100 0x7f05001f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.198+0000 7f0505878700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f050015ded0 0x7f050009e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.198+0000 7f0505878700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f050015e2b0 0x7f05001f6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.198+0000 7f0505878700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050009e590 con 0x7f050015ded0 2024-01-15T11:25:43.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.198+0000 7f0505878700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050009d040 con 0x7f05000d2100 2024-01-15T11:25:43.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.198+0000 7f0505878700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05001f7340 con 0x7f050015e2b0 2024-01-15T11:25:43.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.198+0000 7f04fe7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f050015ded0 0x7f050009e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.198+0000 7f04fe7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f050015ded0 0x7f050009e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59342/0 (socket says 172.21.15.113:59342) 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.198+0000 7f04fe7fc700 1 -- 172.21.15.113:0/1299573805 learned_addr learned my addr 172.21.15.113:0/1299573805 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.198+0000 7f04feffd700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05000d2100 0x7f05001f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.198+0000 7f04ff7fe700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f050015e2b0 0x7f05001f6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.199+0000 7f04fe7fc700 1 -- 172.21.15.113:0/1299573805 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f050015e2b0 msgr2=0x7f05001f6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.199+0000 7f04fe7fc700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f050015e2b0 0x7f05001f6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.199+0000 7f04fe7fc700 1 -- 172.21.15.113:0/1299573805 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05000d2100 msgr2=0x7f05001f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.199+0000 7f04fe7fc700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05000d2100 0x7f05001f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.199+0000 7f04fe7fc700 1 -- 172.21.15.113:0/1299573805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f8009a70 con 0x7f050015ded0 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.199+0000 7f04fe7fc700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f050015ded0 0x7f050009e050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f04f000c4d0 tx=0x7f04f000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.199+0000 7f04e7fff700 1 -- 172.21.15.113:0/1299573805 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f0011790 con 0x7f050015ded0 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.199+0000 7f04e7fff700 1 -- 172.21.15.113:0/1299573805 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f00118f0 con 0x7f050015ded0 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.199+0000 7f0505878700 1 -- 172.21.15.113:0/1299573805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050009cd60 con 0x7f050015ded0 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.199+0000 7f04e7fff700 1 -- 172.21.15.113:0/1299573805 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f0011ca0 con 0x7f050015ded0 2024-01-15T11:25:43.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.199+0000 7f0505878700 1 -- 172.21.15.113:0/1299573805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05000c9110 con 0x7f050015ded0 2024-01-15T11:25:43.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.200+0000 7f0505878700 1 -- 172.21.15.113:0/1299573805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0500069020 con 0x7f050015ded0 2024-01-15T11:25:43.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.200+0000 7f04e7fff700 1 -- 172.21.15.113:0/1299573805 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f04f001c460 con 0x7f050015ded0 2024-01-15T11:25:43.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.201+0000 7f04e7fff700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f04ec06b350 0x7f04ec06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.201+0000 7f04feffd700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f04ec06b350 0x7f04ec06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.201+0000 7f04e7fff700 1 -- 172.21.15.113:0/1299573805 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04f0089a80 con 0x7f050015ded0 2024-01-15T11:25:43.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.201+0000 7f04feffd700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f04ec06b350 0x7f04ec06d810 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f04f4000f60 tx=0x7f04f4007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:43.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.205+0000 7f04e7fff700 1 -- 172.21.15.113:0/1299573805 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04f00548e0 con 0x7f050015ded0 2024-01-15T11:25:43.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.396+0000 7f0505878700 1 -- 172.21.15.113:0/1299573805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f050006b010 con 0x7f050015ded0 2024-01-15T11:25:43.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.397+0000 7f04e7fff700 1 -- 172.21.15.113:0/1299573805 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f04f0057f00 con 0x7f050015ded0 2024-01-15T11:25:43.395 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:43.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.398+0000 7f0505878700 1 -- 172.21.15.113:0/1299573805 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f04ec06b350 msgr2=0x7f04ec06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.398+0000 7f0505878700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f04ec06b350 0x7f04ec06d810 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f04f4000f60 tx=0x7f04f4007c00 comp rx=0 tx=0).stop 2024-01-15T11:25:43.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.398+0000 7f0505878700 1 -- 172.21.15.113:0/1299573805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f050015ded0 msgr2=0x7f050009e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.398+0000 7f0505878700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f050015ded0 0x7f050009e050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f04f000c4d0 tx=0x7f04f000c890 comp rx=0 tx=0).stop 2024-01-15T11:25:43.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.399+0000 7f0505878700 1 -- 172.21.15.113:0/1299573805 shutdown_connections 2024-01-15T11:25:43.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.399+0000 7f0505878700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f04ec06b350 0x7f04ec06d810 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.399+0000 7f0505878700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f050015e2b0 0x7f05001f6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.399+0000 7f0505878700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f05000d2100 0x7f05001f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.399+0000 7f0505878700 1 --2- 172.21.15.113:0/1299573805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f050015ded0 0x7f050009e050 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.399+0000 7f0505878700 1 -- 172.21.15.113:0/1299573805 >> 172.21.15.113:0/1299573805 conn(0x7f050009aa00 msgr2=0x7f05000b0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:43.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.399+0000 7f0505878700 1 -- 172.21.15.113:0/1299573805 shutdown_connections 2024-01-15T11:25:43.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.399+0000 7f0505878700 1 -- 172.21.15.113:0/1299573805 wait complete. 2024-01-15T11:25:43.409 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T11:25:43.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.534+0000 7f2eb532c700 1 Processor -- start 2024-01-15T11:25:43.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.534+0000 7f2eb532c700 1 -- start start 2024-01-15T11:25:43.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eb532c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2eb0161510 0x7f2eb00cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eb532c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2eb00d3710 0x7f2eb00cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eb532c700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb0162af0 con 0x7f2eb00b98e0 2024-01-15T11:25:43.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eb532c700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb00d13e0 con 0x7f2eb00d3710 2024-01-15T11:25:43.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eb532c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb0161f70 con 0x7f2eb0161510 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eaf7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2eb00d3710 0x7f2eb00cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eaf7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2eb00d3710 0x7f2eb00cf160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59358/0 (socket says 172.21.15.113:59358) 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eaf7fe700 1 -- 172.21.15.113:0/551920617 learned_addr learned my addr 172.21.15.113:0/551920617 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eaf7fe700 1 -- 172.21.15.113:0/551920617 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2eb0161510 msgr2=0x7f2eb00cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eaeffd700 1 --2- 172.21.15.113:0/551920617 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2eb0161510 0x7f2eb00cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eaf7fe700 1 --2- 172.21.15.113:0/551920617 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2eb0161510 0x7f2eb00cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eaf7fe700 1 -- 172.21.15.113:0/551920617 >> v1:172.21.15.141:6789/0 conn(0x7f2eb00b98e0 legacy=0x7f2eb00b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eaf7fe700 1 -- 172.21.15.113:0/551920617 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb0162250 con 0x7f2eb00d3710 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.535+0000 7f2eaf7fe700 1 --2- 172.21.15.113:0/551920617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb00d3710 0x7f2eb00cf160 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009e50 tx=0x7f2ea800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88551c507619b828 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.536+0000 7f2eadffb700 1 -- 172.21.15.113:0/551920617 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea800e070 con 0x7f2eb00d3710 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.536+0000 7f2eadffb700 1 -- 172.21.15.113:0/551920617 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea8003a20 con 0x7f2eb00d3710 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.536+0000 7f2eadffb700 1 -- 172.21.15.113:0/551920617 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8003dd0 con 0x7f2eb00d3710 2024-01-15T11:25:43.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.536+0000 7f2eb532c700 1 -- 172.21.15.113:0/551920617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb00d3710 msgr2=0x7f2eb00cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.536+0000 7f2eb532c700 1 --2- 172.21.15.113:0/551920617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb00d3710 0x7f2eb00cf160 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009e50 tx=0x7f2ea800b330 comp rx=0 tx=0).stop 2024-01-15T11:25:43.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.536+0000 7f2eb532c700 1 -- 172.21.15.113:0/551920617 shutdown_connections 2024-01-15T11:25:43.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.536+0000 7f2eb532c700 1 --2- 172.21.15.113:0/551920617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb00d3710 0x7f2eb00cf160 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.536+0000 7f2eb532c700 1 --2- 172.21.15.113:0/551920617 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2eb0161510 0x7f2eb00cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.536+0000 7f2eb532c700 1 -- 172.21.15.113:0/551920617 >> 172.21.15.113:0/551920617 conn(0x7f2eb006b410 msgr2=0x7f2eb006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:43.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.536+0000 7f2eb532c700 1 -- 172.21.15.113:0/551920617 shutdown_connections 2024-01-15T11:25:43.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.536+0000 7f2eb532c700 1 -- 172.21.15.113:0/551920617 wait complete. 2024-01-15T11:25:43.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.537+0000 7f2eb532c700 1 Processor -- start 2024-01-15T11:25:43.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.538+0000 7f2eb532c700 1 -- start start 2024-01-15T11:25:43.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.538+0000 7f2eb532c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb00b98e0 0x7f2eb015ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.538+0000 7f2eb532c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2eb00d3710 0x7f2eb0157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.538+0000 7f2eb532c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2eb0161510 0x7f2eb00b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.538+0000 7f2eb532c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb00b4a10 con 0x7f2eb00b98e0 2024-01-15T11:25:43.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.538+0000 7f2eb532c700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb01579b0 con 0x7f2eb00d3710 2024-01-15T11:25:43.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.538+0000 7f2eb532c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb0159830 con 0x7f2eb0161510 2024-01-15T11:25:43.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.538+0000 7f2eaeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb00b98e0 0x7f2eb015ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.538+0000 7f2eaeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb00b98e0 0x7f2eb015ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59374/0 (socket says 172.21.15.113:59374) 2024-01-15T11:25:43.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.538+0000 7f2eaeffd700 1 -- 172.21.15.113:0/1790826634 learned_addr learned my addr 172.21.15.113:0/1790826634 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2eaf7fe700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2eb0161510 0x7f2eb00b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2eaeffd700 1 -- 172.21.15.113:0/1790826634 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2eb0161510 msgr2=0x7f2eb00b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2eaeffd700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2eb0161510 0x7f2eb00b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2eaeffd700 1 -- 172.21.15.113:0/1790826634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2eb00d3710 msgr2=0x7f2eb0157470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2eae7fc700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2eb00d3710 0x7f2eb0157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2eaeffd700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2eb00d3710 0x7f2eb0157470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2eaeffd700 1 -- 172.21.15.113:0/1790826634 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea8009b30 con 0x7f2eb00b98e0 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2eaeffd700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb00b98e0 0x7f2eb015ac70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400c7f0 tx=0x7f2ea400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2e977fe700 1 -- 172.21.15.113:0/1790826634 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4011800 con 0x7f2eb00b98e0 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2e977fe700 1 -- 172.21.15.113:0/1790826634 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea4011960 con 0x7f2eb00b98e0 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2eb532c700 1 -- 172.21.15.113:0/1790826634 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb00b3ef0 con 0x7f2eb00b98e0 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2e977fe700 1 -- 172.21.15.113:0/1790826634 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4011cf0 con 0x7f2eb00b98e0 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.539+0000 7f2eb532c700 1 -- 172.21.15.113:0/1790826634 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb00b4450 con 0x7f2eb00b98e0 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.540+0000 7f2eb532c700 1 -- 172.21.15.113:0/1790826634 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb0068450 con 0x7f2eb00b98e0 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.540+0000 7f2e977fe700 1 -- 172.21.15.113:0/1790826634 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2ea4018360 con 0x7f2eb00b98e0 2024-01-15T11:25:43.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.541+0000 7f2e977fe700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2e9c06b350 0x7f2e9c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.541+0000 7f2e977fe700 1 -- 172.21.15.113:0/1790826634 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ea4089ab0 con 0x7f2eb00b98e0 2024-01-15T11:25:43.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.541+0000 7f2eae7fc700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2e9c06b350 0x7f2e9c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.541+0000 7f2eae7fc700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2e9c06b350 0x7f2e9c06d810 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0160710 tx=0x7f2ea0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:43.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.545+0000 7f2e977fe700 1 -- 172.21.15.113:0/1790826634 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ea4054920 con 0x7f2eb00b98e0 2024-01-15T11:25:43.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.734+0000 7f2eb532c700 1 -- 172.21.15.113:0/1790826634 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2eb0069020 con 0x7f2eb00b98e0 2024-01-15T11:25:43.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.735+0000 7f2e977fe700 1 -- 172.21.15.113:0/1790826634 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2ea401f020 con 0x7f2eb00b98e0 2024-01-15T11:25:43.734 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-15T11:25:43.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.737+0000 7f2eb532c700 1 -- 172.21.15.113:0/1790826634 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2e9c06b350 msgr2=0x7f2e9c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.737+0000 7f2eb532c700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2e9c06b350 0x7f2e9c06d810 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0160710 tx=0x7f2ea0009040 comp rx=0 tx=0).stop 2024-01-15T11:25:43.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.737+0000 7f2eb532c700 1 -- 172.21.15.113:0/1790826634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb00b98e0 msgr2=0x7f2eb015ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.737+0000 7f2eb532c700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb00b98e0 0x7f2eb015ac70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400c7f0 tx=0x7f2ea400cbb0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.737+0000 7f2eb532c700 1 -- 172.21.15.113:0/1790826634 shutdown_connections 2024-01-15T11:25:43.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.737+0000 7f2eb532c700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2e9c06b350 0x7f2e9c06d810 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.737+0000 7f2eb532c700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2eb0161510 0x7f2eb00b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.737+0000 7f2eb532c700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb00b98e0 0x7f2eb015ac70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.737+0000 7f2eb532c700 1 --2- 172.21.15.113:0/1790826634 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2eb00d3710 0x7f2eb0157470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.737+0000 7f2eb532c700 1 -- 172.21.15.113:0/1790826634 >> 172.21.15.113:0/1790826634 conn(0x7f2eb006b410 msgr2=0x7f2eb015f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:43.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.737+0000 7f2eb532c700 1 -- 172.21.15.113:0/1790826634 shutdown_connections 2024-01-15T11:25:43.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.737+0000 7f2eb532c700 1 -- 172.21.15.113:0/1790826634 wait complete. 2024-01-15T11:25:43.748 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T11:25:43.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.872+0000 7fac54d24700 1 Processor -- start 2024-01-15T11:25:43.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.873+0000 7fac54d24700 1 -- start start 2024-01-15T11:25:43.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.873+0000 7fac54d24700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac50166f20 0x7fac50169310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.873+0000 7fac54d24700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5015e860 con 0x7fac5006c550 2024-01-15T11:25:43.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.873+0000 7fac54d24700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac500cf490 con 0x7fac5015e480 2024-01-15T11:25:43.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.873+0000 7fac54d24700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac500701c0 con 0x7fac50166f20 2024-01-15T11:25:43.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4dd9b700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fac5006c550 0x7fac5006f590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42548/0 (socket says 172.21.15.113:42548) 2024-01-15T11:25:43.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4dd9b700 1 -- 172.21.15.113:0/4224733969 learned_addr learned my addr 172.21.15.113:0/4224733969 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:43.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4ed9d700 1 --2- 172.21.15.113:0/4224733969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac50166f20 0x7fac50169310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837452401 0 0) 0x7fac5015e860 con 0x7fac5006c550 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac380035b0 con 0x7fac5006c550 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1471722038 0 0) 0x7fac380035b0 con 0x7fac5006c550 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac50166f20 msgr2=0x7fac50169310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 --2- 172.21.15.113:0/4224733969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac50166f20 0x7fac50169310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 >> v1:172.21.15.141:6789/0 conn(0x7fac5015e480 legacy=0x7fac5006ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5006fca0 con 0x7fac5006c550 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fac40001670 con 0x7fac5006c550 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 >> v1:172.21.15.113:6789/0 conn(0x7fac5006c550 legacy=0x7fac5006f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 --2- 172.21.15.113:0/4224733969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac38003a30 0x7fac38003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 --2- 172.21.15.113:0/4224733969 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac38004850 0x7fac38004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 --2- 172.21.15.113:0/4224733969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac50166f20 0x7fac38005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac500701c0 con 0x7fac38003a30 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3800bb40 con 0x7fac38004850 2024-01-15T11:25:43.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3800bd00 con 0x7fac50166f20 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4e59c700 1 --2- 172.21.15.113:0/4224733969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac38003a30 0x7fac38003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4dd9b700 1 --2- 172.21.15.113:0/4224733969 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac38004850 0x7fac38004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.874+0000 7fac4ed9d700 1 --2- 172.21.15.113:0/4224733969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac50166f20 0x7fac38005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac4e59c700 1 -- 172.21.15.113:0/4224733969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac50166f20 msgr2=0x7fac38005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac4e59c700 1 --2- 172.21.15.113:0/4224733969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac50166f20 0x7fac38005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac4e59c700 1 -- 172.21.15.113:0/4224733969 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac38004850 msgr2=0x7fac38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac4e59c700 1 --2- 172.21.15.113:0/4224733969 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac38004850 0x7fac38004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac4e59c700 1 -- 172.21.15.113:0/4224733969 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac380035b0 con 0x7fac38003a30 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac4e59c700 1 --2- 172.21.15.113:0/4224733969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac38003a30 0x7fac38003e50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fac440017c0 tx=0x7fac38005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3364ba3369df0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac440052d0 con 0x7fac38003a30 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac44005430 con 0x7fac38003a30 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac4d59a700 1 -- 172.21.15.113:0/4224733969 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac440038c0 con 0x7fac38003a30 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac54d24700 1 -- 172.21.15.113:0/4224733969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac38003a30 msgr2=0x7fac38003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac54d24700 1 --2- 172.21.15.113:0/4224733969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac38003a30 0x7fac38003e50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fac440017c0 tx=0x7fac38005880 comp rx=0 tx=0).stop 2024-01-15T11:25:43.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac4ed9d700 1 -- 172.21.15.113:0/4224733969 reap_dead start 2024-01-15T11:25:43.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac54d24700 1 -- 172.21.15.113:0/4224733969 shutdown_connections 2024-01-15T11:25:43.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac54d24700 1 -- 172.21.15.113:0/4224733969 >> 172.21.15.113:0/4224733969 conn(0x7fac500b37f0 msgr2=0x7fac500b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:43.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.875+0000 7fac54d24700 1 -- 172.21.15.113:0/4224733969 shutdown_connections 2024-01-15T11:25:43.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.876+0000 7fac54d24700 1 -- 172.21.15.113:0/4224733969 wait complete. 2024-01-15T11:25:43.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.877+0000 7fac54d24700 1 Processor -- start 2024-01-15T11:25:43.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.877+0000 7fac54d24700 1 -- start start 2024-01-15T11:25:43.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac54d24700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac38003a30 0x7fac5014c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac54d24700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac50157b80 0x7fac50158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac54d24700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac500ca820 0x7fac500caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac54d24700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5006f540 con 0x7fac500ca820 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac54d24700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac50158540 con 0x7fac50157b80 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac54d24700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5015f4a0 con 0x7fac38003a30 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac4ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac500ca820 0x7fac500caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac4ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac500ca820 0x7fac500caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59388/0 (socket says 172.21.15.113:59388) 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac4ed9d700 1 -- 172.21.15.113:0/436671605 learned_addr learned my addr 172.21.15.113:0/436671605 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac4e59c700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac38003a30 0x7fac5014c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac4dd9b700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac50157b80 0x7fac50158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac4ed9d700 1 -- 172.21.15.113:0/436671605 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac38003a30 msgr2=0x7fac5014c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac4ed9d700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac38003a30 0x7fac5014c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac4ed9d700 1 -- 172.21.15.113:0/436671605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac50157b80 msgr2=0x7fac50158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac4ed9d700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac50157b80 0x7fac50158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:43.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac4ed9d700 1 -- 172.21.15.113:0/436671605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5014c610 con 0x7fac500ca820 2024-01-15T11:25:43.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac4ed9d700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac500ca820 0x7fac500caca0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fac500d1bb0 tx=0x7fac48011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:43.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac3effd700 1 -- 172.21.15.113:0/436671605 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4800e040 con 0x7fac500ca820 2024-01-15T11:25:43.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac3effd700 1 -- 172.21.15.113:0/436671605 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4800f040 con 0x7fac500ca820 2024-01-15T11:25:43.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac3effd700 1 -- 172.21.15.113:0/436671605 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4800f880 con 0x7fac500ca820 2024-01-15T11:25:43.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac54d24700 1 -- 172.21.15.113:0/436671605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5015e900 con 0x7fac500ca820 2024-01-15T11:25:43.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.878+0000 7fac54d24700 1 -- 172.21.15.113:0/436671605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5015eb70 con 0x7fac500ca820 2024-01-15T11:25:43.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.879+0000 7fac54d24700 1 -- 172.21.15.113:0/436671605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac50069020 con 0x7fac500ca820 2024-01-15T11:25:43.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.880+0000 7fac3effd700 1 -- 172.21.15.113:0/436671605 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fac4800f9e0 con 0x7fac500ca820 2024-01-15T11:25:43.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.880+0000 7fac3effd700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fac3006b350 0x7fac3006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:43.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.880+0000 7fac3effd700 1 -- 172.21.15.113:0/436671605 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac48089f10 con 0x7fac500ca820 2024-01-15T11:25:43.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.880+0000 7fac4e59c700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fac3006b350 0x7fac3006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:43.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.881+0000 7fac4e59c700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fac3006b350 0x7fac3006d810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fac44000c00 tx=0x7fac44009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:43.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:43.885+0000 7fac3effd700 1 -- 172.21.15.113:0/436671605 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac48054d80 con 0x7fac500ca820 2024-01-15T11:25:44.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.074+0000 7fac54d24700 1 -- 172.21.15.113:0/436671605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fac500d1970 con 0x7fac500ca820 2024-01-15T11:25:44.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.074+0000 7fac3effd700 1 -- 172.21.15.113:0/436671605 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fac48013310 con 0x7fac500ca820 2024-01-15T11:25:44.073 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-15T11:25:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.077+0000 7fac54d24700 1 -- 172.21.15.113:0/436671605 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fac3006b350 msgr2=0x7fac3006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.077+0000 7fac54d24700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fac3006b350 0x7fac3006d810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fac44000c00 tx=0x7fac44009030 comp rx=0 tx=0).stop 2024-01-15T11:25:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.077+0000 7fac54d24700 1 -- 172.21.15.113:0/436671605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac500ca820 msgr2=0x7fac500caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.077+0000 7fac54d24700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac500ca820 0x7fac500caca0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fac500d1bb0 tx=0x7fac48011f60 comp rx=0 tx=0).stop 2024-01-15T11:25:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.077+0000 7fac54d24700 1 -- 172.21.15.113:0/436671605 shutdown_connections 2024-01-15T11:25:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.077+0000 7fac54d24700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fac3006b350 0x7fac3006d810 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.077+0000 7fac54d24700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac500ca820 0x7fac500caca0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.077+0000 7fac54d24700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac38003a30 0x7fac5014c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.077+0000 7fac54d24700 1 --2- 172.21.15.113:0/436671605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac50157b80 0x7fac50158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.077+0000 7fac54d24700 1 -- 172.21.15.113:0/436671605 >> 172.21.15.113:0/436671605 conn(0x7fac500b37f0 msgr2=0x7fac500b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.077+0000 7fac54d24700 1 -- 172.21.15.113:0/436671605 shutdown_connections 2024-01-15T11:25:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.077+0000 7fac54d24700 1 -- 172.21.15.113:0/436671605 wait complete. 2024-01-15T11:25:44.088 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:44.091 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:25:44.133 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:25:44.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.214+0000 7fb2a50f4700 1 Processor -- start 2024-01-15T11:25:44.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.214+0000 7fb2a50f4700 1 -- start start 2024-01-15T11:25:44.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.214+0000 7fb2a50f4700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb2a006ee50 0x7fb2a0074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.214+0000 7fb2a50f4700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb2a015c140 0x7fb2a00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.214+0000 7fb2a50f4700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a0074e90 con 0x7fb2a0073f90 2024-01-15T11:25:44.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.214+0000 7fb2a50f4700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a00748b0 con 0x7fb2a006ee50 2024-01-15T11:25:44.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.214+0000 7fb2a50f4700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a015c700 con 0x7fb2a015c140 2024-01-15T11:25:44.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.215+0000 7fb29ed9d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb2a0073f90 0x7fb2a00ce090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42550/0 (socket says 172.21.15.113:42550) 2024-01-15T11:25:44.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.215+0000 7fb29ed9d700 1 -- 172.21.15.113:0/1578000259 learned_addr learned my addr 172.21.15.113:0/1578000259 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:44.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.215+0000 7fb29e59c700 1 --2- 172.21.15.113:0/1578000259 >> v2:172.21.15.113:3300/0 conn(0x7fb2a006ee50 0x7fb2a0074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.215+0000 7fb29f59e700 1 --2- 172.21.15.113:0/1578000259 >> v2:172.21.15.141:3300/0 conn(0x7fb2a015c140 0x7fb2a00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.215+0000 7fb29e59c700 1 -- 172.21.15.113:0/1578000259 >> v2:172.21.15.141:3300/0 conn(0x7fb2a015c140 msgr2=0x7fb2a00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:44.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.215+0000 7fb29e59c700 1 --2- 172.21.15.113:0/1578000259 >> v2:172.21.15.141:3300/0 conn(0x7fb2a015c140 0x7fb2a00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.215+0000 7fb29e59c700 1 -- 172.21.15.113:0/1578000259 >> v1:172.21.15.113:6789/0 conn(0x7fb2a0073f90 legacy=0x7fb2a00ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.215+0000 7fb29e59c700 1 -- 172.21.15.113:0/1578000259 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a015c9e0 con 0x7fb2a006ee50 2024-01-15T11:25:44.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.215+0000 7fb29e59c700 1 --2- 172.21.15.113:0/1578000259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a006ee50 0x7fb2a0074370 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb294009840 tx=0x7fb294009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a7889dab56bad2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:44.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.216+0000 7fb29dd9b700 1 -- 172.21.15.113:0/1578000259 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294011830 con 0x7fb2a006ee50 2024-01-15T11:25:44.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.216+0000 7fb29dd9b700 1 -- 172.21.15.113:0/1578000259 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb294011990 con 0x7fb2a006ee50 2024-01-15T11:25:44.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.216+0000 7fb29dd9b700 1 -- 172.21.15.113:0/1578000259 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294011d20 con 0x7fb2a006ee50 2024-01-15T11:25:44.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.217+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1578000259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a006ee50 msgr2=0x7fb2a0074370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.217+0000 7fb2a50f4700 1 --2- 172.21.15.113:0/1578000259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a006ee50 0x7fb2a0074370 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb294009840 tx=0x7fb294009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:44.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.217+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1578000259 shutdown_connections 2024-01-15T11:25:44.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.217+0000 7fb2a50f4700 1 --2- 172.21.15.113:0/1578000259 >> v2:172.21.15.141:3300/0 conn(0x7fb2a015c140 0x7fb2a00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.217+0000 7fb2a50f4700 1 --2- 172.21.15.113:0/1578000259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a006ee50 0x7fb2a0074370 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.217+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1578000259 >> 172.21.15.113:0/1578000259 conn(0x7fb2a00cd570 msgr2=0x7fb2a00cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:44.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.217+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1578000259 shutdown_connections 2024-01-15T11:25:44.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.217+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1578000259 wait complete. 2024-01-15T11:25:44.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.218+0000 7fb2a50f4700 1 Processor -- start 2024-01-15T11:25:44.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.219+0000 7fb2a50f4700 1 -- start start 2024-01-15T11:25:44.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.219+0000 7fb2a50f4700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb2a006ee50 0x7fb2a014a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.219+0000 7fb2a50f4700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2a0073f90 0x7fb2a01e7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.219+0000 7fb2a50f4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a015c140 0x7fb2a0153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.219+0000 7fb2a50f4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a015f590 con 0x7fb2a015c140 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.219+0000 7fb2a50f4700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a014abf0 con 0x7fb2a0073f90 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.219+0000 7fb2a50f4700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a015e9c0 con 0x7fb2a006ee50 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.219+0000 7fb29f59e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a015c140 0x7fb2a0153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.219+0000 7fb29f59e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a015c140 0x7fb2a0153ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59392/0 (socket says 172.21.15.113:59392) 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.219+0000 7fb29f59e700 1 -- 172.21.15.113:0/1844097717 learned_addr learned my addr 172.21.15.113:0/1844097717 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.219+0000 7fb29ed9d700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb2a006ee50 0x7fb2a014a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.220+0000 7fb29e59c700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2a0073f90 0x7fb2a01e7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.220+0000 7fb29f59e700 1 -- 172.21.15.113:0/1844097717 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb2a006ee50 msgr2=0x7fb2a014a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.220+0000 7fb29f59e700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb2a006ee50 0x7fb2a014a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.220+0000 7fb29f59e700 1 -- 172.21.15.113:0/1844097717 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2a0073f90 msgr2=0x7fb2a01e7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.220+0000 7fb29f59e700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2a0073f90 0x7fb2a01e7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.220+0000 7fb29f59e700 1 -- 172.21.15.113:0/1844097717 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2940094a0 con 0x7fb2a015c140 2024-01-15T11:25:44.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.220+0000 7fb29f59e700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a015c140 0x7fb2a0153ce0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb29800efc0 tx=0x7fb29800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:44.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.220+0000 7fb2877fe700 1 -- 172.21.15.113:0/1844097717 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29800cad0 con 0x7fb2a015c140 2024-01-15T11:25:44.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.220+0000 7fb2877fe700 1 -- 172.21.15.113:0/1844097717 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29800f040 con 0x7fb2a015c140 2024-01-15T11:25:44.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.220+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1844097717 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a015ec60 con 0x7fb2a015c140 2024-01-15T11:25:44.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.220+0000 7fb2877fe700 1 -- 172.21.15.113:0/1844097717 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29800ce70 con 0x7fb2a015c140 2024-01-15T11:25:44.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.220+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1844097717 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a015f220 con 0x7fb2a015c140 2024-01-15T11:25:44.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.221+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1844097717 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a0067f60 con 0x7fb2a015c140 2024-01-15T11:25:44.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.222+0000 7fb2877fe700 1 -- 172.21.15.113:0/1844097717 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb298014480 con 0x7fb2a015c140 2024-01-15T11:25:44.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.222+0000 7fb2877fe700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb28c06b350 0x7fb28c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.222+0000 7fb2877fe700 1 -- 172.21.15.113:0/1844097717 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb29808a2e0 con 0x7fb2a015c140 2024-01-15T11:25:44.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.222+0000 7fb29ed9d700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb28c06b350 0x7fb28c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.222+0000 7fb29ed9d700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb28c06b350 0x7fb28c06d810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb290000f60 tx=0x7fb290007f10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:44.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.226+0000 7fb2877fe700 1 -- 172.21.15.113:0/1844097717 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb298055150 con 0x7fb2a015c140 2024-01-15T11:25:44.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.418+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1844097717 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2a015f6d0 con 0x7fb2a015c140 2024-01-15T11:25:44.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.419+0000 7fb2877fe700 1 -- 172.21.15.113:0/1844097717 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7fb298058770 con 0x7fb2a015c140 2024-01-15T11:25:44.418 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:44.419 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.421+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1844097717 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb28c06b350 msgr2=0x7fb28c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.421+0000 7fb2a50f4700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb28c06b350 0x7fb28c06d810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb290000f60 tx=0x7fb290007f10 comp rx=0 tx=0).stop 2024-01-15T11:25:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.421+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1844097717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a015c140 msgr2=0x7fb2a0153ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.421+0000 7fb2a50f4700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a015c140 0x7fb2a0153ce0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb29800efc0 tx=0x7fb29800c0c0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.421+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1844097717 shutdown_connections 2024-01-15T11:25:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.421+0000 7fb2a50f4700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb28c06b350 0x7fb28c06d810 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.421+0000 7fb2a50f4700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a015c140 0x7fb2a0153ce0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.421+0000 7fb2a50f4700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb2a006ee50 0x7fb2a014a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.421+0000 7fb2a50f4700 1 --2- 172.21.15.113:0/1844097717 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb2a0073f90 0x7fb2a01e7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.421+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1844097717 >> 172.21.15.113:0/1844097717 conn(0x7fb2a00cd570 msgr2=0x7fb2a006e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.422+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1844097717 shutdown_connections 2024-01-15T11:25:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.422+0000 7fb2a50f4700 1 -- 172.21.15.113:0/1844097717 wait complete. 2024-01-15T11:25:44.433 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T11:25:13.398220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T11:25:14.589439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:44.433 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:44.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.597+0000 7f4677ff6700 1 Processor -- start 2024-01-15T11:25:44.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.598+0000 7f4677ff6700 1 -- start start 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.598+0000 7f4677ff6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f46700d05b0 0x7f467006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.598+0000 7f4677ff6700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46700b3d60 con 0x7f467015ae00 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.598+0000 7f4677ff6700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46700b4040 con 0x7f46700add40 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.598+0000 7f4677ff6700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f467015f8b0 con 0x7f46700d05b0 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.599+0000 7f4675591700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f46700d05b0 0x7f467006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.599+0000 7f4675d92700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f467015ae00 0x7f46700cfea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42224/0 (socket says 172.21.15.113:42224) 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.599+0000 7f4675d92700 1 -- 172.21.15.113:0/3344152704 learned_addr learned my addr 172.21.15.113:0/3344152704 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.599+0000 7f4675591700 1 -- 172.21.15.113:0/3344152704 >> v1:172.21.15.141:6789/0 conn(0x7f46700add40 legacy=0x7f46700b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.599+0000 7f4675591700 1 -- 172.21.15.113:0/3344152704 >> v1:172.21.15.113:6789/0 conn(0x7f467015ae00 legacy=0x7f46700cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.599+0000 7f4675591700 1 -- 172.21.15.113:0/3344152704 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46700b5be0 con 0x7f46700d05b0 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.599+0000 7f4674d90700 1 -- 172.21.15.113:0/3344152704 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579048108 0 0) 0x7f46700b3d60 con 0x7f467015ae00 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.599+0000 7f4675591700 1 --2- 172.21.15.113:0/3344152704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46700d05b0 0x7f467006dc60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f466c009840 tx=0x7f466c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307162728627c196 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.599+0000 7f4674d90700 1 -- 172.21.15.113:0/3344152704 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c004230 con 0x7f46700d05b0 2024-01-15T11:25:44.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.599+0000 7f4674d90700 1 -- 172.21.15.113:0/3344152704 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c004390 con 0x7f46700d05b0 2024-01-15T11:25:44.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.599+0000 7f4674d90700 1 -- 172.21.15.113:0/3344152704 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c0036a0 con 0x7f46700d05b0 2024-01-15T11:25:44.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.600+0000 7f4677ff6700 1 -- 172.21.15.113:0/3344152704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46700d05b0 msgr2=0x7f467006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.600+0000 7f4677ff6700 1 --2- 172.21.15.113:0/3344152704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46700d05b0 0x7f467006dc60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f466c009840 tx=0x7f466c009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:44.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.600+0000 7f4677ff6700 1 -- 172.21.15.113:0/3344152704 shutdown_connections 2024-01-15T11:25:44.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.600+0000 7f4677ff6700 1 --2- 172.21.15.113:0/3344152704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46700d05b0 0x7f467006dc60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.600+0000 7f4677ff6700 1 -- 172.21.15.113:0/3344152704 >> 172.21.15.113:0/3344152704 conn(0x7f4670072b20 msgr2=0x7f467006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:44.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.600+0000 7f4677ff6700 1 -- 172.21.15.113:0/3344152704 shutdown_connections 2024-01-15T11:25:44.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.600+0000 7f4677ff6700 1 -- 172.21.15.113:0/3344152704 wait complete. 2024-01-15T11:25:44.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.601+0000 7f4677ff6700 1 Processor -- start 2024-01-15T11:25:44.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.601+0000 7f4677ff6700 1 -- start start 2024-01-15T11:25:44.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4677ff6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46700add40 0x7f467009c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4677ff6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46700d05b0 0x7f4670159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4677ff6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f467015ae00 0x7f4670073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4677ff6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4670146e90 con 0x7f467015ae00 2024-01-15T11:25:44.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4677ff6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467009df80 con 0x7f46700add40 2024-01-15T11:25:44.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4677ff6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467015cb20 con 0x7f46700d05b0 2024-01-15T11:25:44.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4676593700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f467015ae00 0x7f4670073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4676593700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f467015ae00 0x7f4670073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44718/0 (socket says 172.21.15.113:44718) 2024-01-15T11:25:44.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4676593700 1 -- 172.21.15.113:0/2809011094 learned_addr learned my addr 172.21.15.113:0/2809011094 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:44.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4675d92700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46700add40 0x7f467009c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4676593700 1 -- 172.21.15.113:0/2809011094 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46700d05b0 msgr2=0x7f4670159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4675591700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46700d05b0 0x7f4670159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4676593700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46700d05b0 0x7f4670159890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4676593700 1 -- 172.21.15.113:0/2809011094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46700add40 msgr2=0x7f467009c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4676593700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46700add40 0x7f467009c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4676593700 1 -- 172.21.15.113:0/2809011094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c0094a0 con 0x7f467015ae00 2024-01-15T11:25:44.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4675d92700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46700add40 0x7f467009c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:44.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4676593700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f467015ae00 0x7f4670073f90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f466800cba0 tx=0x7f466800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:44.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f46667fc700 1 -- 172.21.15.113:0/2809011094 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466800ab80 con 0x7f467015ae00 2024-01-15T11:25:44.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f46667fc700 1 -- 172.21.15.113:0/2809011094 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466800ace0 con 0x7f467015ae00 2024-01-15T11:25:44.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f46667fc700 1 -- 172.21.15.113:0/2809011094 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668012660 con 0x7f467015ae00 2024-01-15T11:25:44.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.602+0000 7f4677ff6700 1 -- 172.21.15.113:0/2809011094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467015ce00 con 0x7f467015ae00 2024-01-15T11:25:44.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.603+0000 7f4677ff6700 1 -- 172.21.15.113:0/2809011094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467015d3c0 con 0x7f467015ae00 2024-01-15T11:25:44.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.603+0000 7f4677ff6700 1 -- 172.21.15.113:0/2809011094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4670069020 con 0x7f467015ae00 2024-01-15T11:25:44.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.604+0000 7f46667fc700 1 -- 172.21.15.113:0/2809011094 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f46680127c0 con 0x7f467015ae00 2024-01-15T11:25:44.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.604+0000 7f46667fc700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f465406b350 0x7f465406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.604+0000 7f4675d92700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f465406b350 0x7f465406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.604+0000 7f46667fc700 1 -- 172.21.15.113:0/2809011094 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f466808a340 con 0x7f467015ae00 2024-01-15T11:25:44.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.605+0000 7f4675d92700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f465406b350 0x7f465406d810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4660007fd0 tx=0x7f4660009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:44.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.609+0000 7f46667fc700 1 -- 172.21.15.113:0/2809011094 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46680551b0 con 0x7f467015ae00 2024-01-15T11:25:44.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.798+0000 7f4677ff6700 1 -- 172.21.15.113:0/2809011094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f467006c8c0 con 0x7f467015ae00 2024-01-15T11:25:44.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.799+0000 7f46667fc700 1 -- 172.21.15.113:0/2809011094 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4668016020 con 0x7f467015ae00 2024-01-15T11:25:44.798 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:44.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.801+0000 7f4677ff6700 1 -- 172.21.15.113:0/2809011094 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f465406b350 msgr2=0x7f465406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.801+0000 7f4677ff6700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f465406b350 0x7f465406d810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4660007fd0 tx=0x7f4660009040 comp rx=0 tx=0).stop 2024-01-15T11:25:44.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.801+0000 7f4677ff6700 1 -- 172.21.15.113:0/2809011094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f467015ae00 msgr2=0x7f4670073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.801+0000 7f4677ff6700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f467015ae00 0x7f4670073f90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f466800cba0 tx=0x7f466800cf60 comp rx=0 tx=0).stop 2024-01-15T11:25:44.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.801+0000 7f4677ff6700 1 -- 172.21.15.113:0/2809011094 shutdown_connections 2024-01-15T11:25:44.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.801+0000 7f4677ff6700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f465406b350 0x7f465406d810 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.801+0000 7f4677ff6700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f467015ae00 0x7f4670073f90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.801+0000 7f4677ff6700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46700add40 0x7f467009c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.801+0000 7f4677ff6700 1 --2- 172.21.15.113:0/2809011094 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46700d05b0 0x7f4670159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.801+0000 7f4677ff6700 1 -- 172.21.15.113:0/2809011094 >> 172.21.15.113:0/2809011094 conn(0x7f4670072b20 msgr2=0x7f46700b5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:44.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.801+0000 7f4677ff6700 1 -- 172.21.15.113:0/2809011094 shutdown_connections 2024-01-15T11:25:44.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.801+0000 7f4677ff6700 1 -- 172.21.15.113:0/2809011094 wait complete. 2024-01-15T11:25:44.812 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T11:25:44.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.936+0000 7f2862ea0700 1 Processor -- start 2024-01-15T11:25:44.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.936+0000 7f2862ea0700 1 -- start start 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.937+0000 7f2862ea0700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f285c147880 0x7f285c06c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.937+0000 7f2862ea0700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f285c0b4e80 0x7f285c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.937+0000 7f2862ea0700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285c147c60 con 0x7f285c06c540 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.937+0000 7f2862ea0700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285c1625a0 con 0x7f285c0b4e80 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.937+0000 7f2862ea0700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285c06c9c0 con 0x7f285c147880 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.937+0000 7f2860c3c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f285c147880 0x7f285c06c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.937+0000 7f2860c3c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f285c147880 0x7f285c06c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:49606/0 (socket says 172.21.15.113:49606) 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.937+0000 7f2860c3c700 1 -- 172.21.15.113:0/1487571725 learned_addr learned my addr 172.21.15.113:0/1487571725 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.937+0000 7f286143d700 1 --2- 172.21.15.113:0/1487571725 >> v2:172.21.15.141:3300/0 conn(0x7f285c0b4e80 0x7f285c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.938+0000 7f2860c3c700 1 -- 172.21.15.113:0/1487571725 >> v2:172.21.15.141:3300/0 conn(0x7f285c0b4e80 msgr2=0x7f285c06e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.938+0000 7f2860c3c700 1 --2- 172.21.15.113:0/1487571725 >> v2:172.21.15.141:3300/0 conn(0x7f285c0b4e80 0x7f285c06e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.938+0000 7f2860c3c700 1 -- 172.21.15.113:0/1487571725 >> v1:172.21.15.141:6789/0 conn(0x7f285c06c540 legacy=0x7f285c0b4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.938+0000 7f2860c3c700 1 -- 172.21.15.113:0/1487571725 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c1629c0 con 0x7f285c147880 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.938+0000 7f285b7fe700 1 -- 172.21.15.113:0/1487571725 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801628395 0 0) 0x7f285c147c60 con 0x7f285c06c540 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.938+0000 7f2860c3c700 1 --2- 172.21.15.113:0/1487571725 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f285c147880 0x7f285c06c000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f28500098f0 tx=0x7f2850009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1331b01179762ee6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.938+0000 7f285b7fe700 1 -- 172.21.15.113:0/1487571725 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850011830 con 0x7f285c147880 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.938+0000 7f285b7fe700 1 -- 172.21.15.113:0/1487571725 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2850011990 con 0x7f285c147880 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.938+0000 7f285b7fe700 1 -- 172.21.15.113:0/1487571725 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850011d20 con 0x7f285c147880 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.939+0000 7f2862ea0700 1 -- 172.21.15.113:0/1487571725 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f285c147880 msgr2=0x7f285c06c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.939+0000 7f2862ea0700 1 --2- 172.21.15.113:0/1487571725 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f285c147880 0x7f285c06c000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f28500098f0 tx=0x7f2850009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.939+0000 7f2862ea0700 1 -- 172.21.15.113:0/1487571725 shutdown_connections 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.939+0000 7f2862ea0700 1 --2- 172.21.15.113:0/1487571725 >> v2:172.21.15.141:3300/0 conn(0x7f285c0b4e80 0x7f285c06e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.939+0000 7f2862ea0700 1 --2- 172.21.15.113:0/1487571725 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f285c147880 0x7f285c06c000 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.939+0000 7f2862ea0700 1 -- 172.21.15.113:0/1487571725 >> 172.21.15.113:0/1487571725 conn(0x7f285c071c10 msgr2=0x7f285c071ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.939+0000 7f2862ea0700 1 -- 172.21.15.113:0/1487571725 shutdown_connections 2024-01-15T11:25:44.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.939+0000 7f2862ea0700 1 -- 172.21.15.113:0/1487571725 wait complete. 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.940+0000 7f2862ea0700 1 Processor -- start 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.940+0000 7f2862ea0700 1 -- start start 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.941+0000 7f2862ea0700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f285c06c540 0x7f285c133960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.941+0000 7f2862ea0700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f285c0b4e80 0x7f285c1579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.941+0000 7f2862ea0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f285c147880 0x7f285c159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.941+0000 7f2862ea0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c1590e0 con 0x7f285c147880 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.941+0000 7f2862ea0700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c159570 con 0x7f285c0b4e80 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.941+0000 7f2862ea0700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285c1584d0 con 0x7f285c06c540 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f286143d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f285c147880 0x7f285c159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f286143d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f285c147880 0x7f285c159800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44722/0 (socket says 172.21.15.113:44722) 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f286143d700 1 -- 172.21.15.113:0/2770319195 learned_addr learned my addr 172.21.15.113:0/2770319195 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f2860c3c700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f285c06c540 0x7f285c133960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f285bfff700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f285c0b4e80 0x7f285c1579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f286143d700 1 -- 172.21.15.113:0/2770319195 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f285c06c540 msgr2=0x7f285c133960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f286143d700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f285c06c540 0x7f285c133960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f286143d700 1 -- 172.21.15.113:0/2770319195 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f285c0b4e80 msgr2=0x7f285c1579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f286143d700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f285c0b4e80 0x7f285c1579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f286143d700 1 -- 172.21.15.113:0/2770319195 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2850009550 con 0x7f285c147880 2024-01-15T11:25:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f286143d700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f285c147880 0x7f285c159800 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f285400efc0 tx=0x7f285400c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f28597fa700 1 -- 172.21.15.113:0/2770319195 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285400cad0 con 0x7f285c147880 2024-01-15T11:25:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f28597fa700 1 -- 172.21.15.113:0/2770319195 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285400f040 con 0x7f285c147880 2024-01-15T11:25:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f2862ea0700 1 -- 172.21.15.113:0/2770319195 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285c1587b0 con 0x7f285c147880 2024-01-15T11:25:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f28597fa700 1 -- 172.21.15.113:0/2770319195 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285400ce70 con 0x7f285c147880 2024-01-15T11:25:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.942+0000 7f2862ea0700 1 -- 172.21.15.113:0/2770319195 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285c158d70 con 0x7f285c147880 2024-01-15T11:25:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.943+0000 7f2862ea0700 1 -- 172.21.15.113:0/2770319195 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285c159290 con 0x7f285c147880 2024-01-15T11:25:44.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.944+0000 7f28597fa700 1 -- 172.21.15.113:0/2770319195 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2854014480 con 0x7f285c147880 2024-01-15T11:25:44.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.944+0000 7f28597fa700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f284006b350 0x7f284006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:44.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.944+0000 7f28597fa700 1 -- 172.21.15.113:0/2770319195 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f285408a5c0 con 0x7f285c147880 2024-01-15T11:25:44.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.944+0000 7f2860c3c700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f284006b350 0x7f284006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:44.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.945+0000 7f2860c3c700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f284006b350 0x7f284006d810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2850014040 tx=0x7f2850008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:44.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:44.949+0000 7f28597fa700 1 -- 172.21.15.113:0/2770319195 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f285c159290 con 0x7f285c147880 2024-01-15T11:25:45.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.138+0000 7f2862ea0700 1 -- 172.21.15.113:0/2770319195 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f285c069020 con 0x7f285c147880 2024-01-15T11:25:45.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.138+0000 7f28597fa700 1 -- 172.21.15.113:0/2770319195 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2854055430 con 0x7f285c147880 2024-01-15T11:25:45.137 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-15T11:25:45.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.140+0000 7f2862ea0700 1 -- 172.21.15.113:0/2770319195 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f284006b350 msgr2=0x7f284006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.140+0000 7f2862ea0700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f284006b350 0x7f284006d810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2850014040 tx=0x7f2850008030 comp rx=0 tx=0).stop 2024-01-15T11:25:45.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.140+0000 7f2862ea0700 1 -- 172.21.15.113:0/2770319195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f285c147880 msgr2=0x7f285c159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.140+0000 7f2862ea0700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f285c147880 0x7f285c159800 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f285400efc0 tx=0x7f285400c0c0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.141+0000 7f2862ea0700 1 -- 172.21.15.113:0/2770319195 shutdown_connections 2024-01-15T11:25:45.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.141+0000 7f2862ea0700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f284006b350 0x7f284006d810 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.141+0000 7f2862ea0700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f285c147880 0x7f285c159800 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.141+0000 7f2862ea0700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f285c06c540 0x7f285c133960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.141+0000 7f2862ea0700 1 --2- 172.21.15.113:0/2770319195 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f285c0b4e80 0x7f285c1579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.141+0000 7f2862ea0700 1 -- 172.21.15.113:0/2770319195 >> 172.21.15.113:0/2770319195 conn(0x7f285c071c10 msgr2=0x7f285c073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:45.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.141+0000 7f2862ea0700 1 -- 172.21.15.113:0/2770319195 shutdown_connections 2024-01-15T11:25:45.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.141+0000 7f2862ea0700 1 -- 172.21.15.113:0/2770319195 wait complete. 2024-01-15T11:25:45.151 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T11:25:45.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.278+0000 7fa9c922a700 1 Processor -- start 2024-01-15T11:25:45.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.278+0000 7fa9c922a700 1 -- start start 2024-01-15T11:25:45.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.279+0000 7fa9c922a700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa9c406c2e0 0x7fa9c415b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.279+0000 7fa9c922a700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa9c415bb60 0x7fa9c41647f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.279+0000 7fa9c922a700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c406eeb0 con 0x7fa9c415b240 2024-01-15T11:25:45.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.279+0000 7fa9c922a700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c406f190 con 0x7fa9c415bb60 2024-01-15T11:25:45.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.279+0000 7fa9c922a700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c406f470 con 0x7fa9c406c2e0 2024-01-15T11:25:45.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.279+0000 7fa9c259c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa9c406c2e0 0x7fa9c415b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.279+0000 7fa9c359e700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa9c415bb60 0x7fa9c41647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.279+0000 7fa9c359e700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa9c415bb60 0x7fa9c41647f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:50068/0 (socket says 172.21.15.113:50068) 2024-01-15T11:25:45.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.279+0000 7fa9c2d9d700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fa9c415b240 0x7fa9c406e460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:55122/0 (socket says 172.21.15.113:55122) 2024-01-15T11:25:45.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.279+0000 7fa9c359e700 1 -- 172.21.15.113:0/2895751725 learned_addr learned my addr 172.21.15.113:0/2895751725 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:45.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.280+0000 7fa9c259c700 1 -- 172.21.15.113:0/2895751725 >> v2:172.21.15.141:3300/0 conn(0x7fa9c415bb60 msgr2=0x7fa9c41647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:45.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.280+0000 7fa9c259c700 1 --2- 172.21.15.113:0/2895751725 >> v2:172.21.15.141:3300/0 conn(0x7fa9c415bb60 0x7fa9c41647f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.280+0000 7fa9c259c700 1 -- 172.21.15.113:0/2895751725 >> v1:172.21.15.141:6789/0 conn(0x7fa9c415b240 legacy=0x7fa9c406e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.280+0000 7fa9c259c700 1 -- 172.21.15.113:0/2895751725 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c406f750 con 0x7fa9c406c2e0 2024-01-15T11:25:45.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.280+0000 7fa9c359e700 1 --2- 172.21.15.113:0/2895751725 >> v2:172.21.15.141:3300/0 conn(0x7fa9c415bb60 0x7fa9c41647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:45.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.280+0000 7fa9c259c700 1 --2- 172.21.15.113:0/2895751725 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa9c406c2e0 0x7fa9c415b620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009840 tx=0x7fa9b4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d05e10c05dde98b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:45.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.280+0000 7fa9c1d9b700 1 -- 172.21.15.113:0/2895751725 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4011830 con 0x7fa9c406c2e0 2024-01-15T11:25:45.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.280+0000 7fa9c1d9b700 1 -- 172.21.15.113:0/2895751725 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b4011990 con 0x7fa9c406c2e0 2024-01-15T11:25:45.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.280+0000 7fa9c1d9b700 1 -- 172.21.15.113:0/2895751725 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4011d20 con 0x7fa9c406c2e0 2024-01-15T11:25:45.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.280+0000 7fa9c922a700 1 -- 172.21.15.113:0/2895751725 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa9c406c2e0 msgr2=0x7fa9c415b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.280+0000 7fa9c922a700 1 --2- 172.21.15.113:0/2895751725 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa9c406c2e0 0x7fa9c415b620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009840 tx=0x7fa9b4009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:45.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.281+0000 7fa9c922a700 1 -- 172.21.15.113:0/2895751725 shutdown_connections 2024-01-15T11:25:45.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.281+0000 7fa9c922a700 1 --2- 172.21.15.113:0/2895751725 >> v2:172.21.15.141:3300/0 conn(0x7fa9c415bb60 0x7fa9c41647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.281+0000 7fa9c922a700 1 --2- 172.21.15.113:0/2895751725 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa9c406c2e0 0x7fa9c415b620 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.281+0000 7fa9c922a700 1 -- 172.21.15.113:0/2895751725 >> 172.21.15.113:0/2895751725 conn(0x7fa9c40d0c00 msgr2=0x7fa9c40d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:45.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.281+0000 7fa9c922a700 1 -- 172.21.15.113:0/2895751725 shutdown_connections 2024-01-15T11:25:45.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.281+0000 7fa9c922a700 1 -- 172.21.15.113:0/2895751725 wait complete. 2024-01-15T11:25:45.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.282+0000 7fa9c922a700 1 Processor -- start 2024-01-15T11:25:45.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.282+0000 7fa9c922a700 1 -- start start 2024-01-15T11:25:45.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c922a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9c406c2e0 0x7fa9c4154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c922a700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa9c415b240 0x7fa9c41f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c922a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9c415bb60 0x7fa9c41f5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c922a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c40cef40 con 0x7fa9c406c2e0 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c922a700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c40cf190 con 0x7fa9c415bb60 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c922a700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c40cf470 con 0x7fa9c415b240 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c2d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9c406c2e0 0x7fa9c4154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c2d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9c406c2e0 0x7fa9c4154020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44734/0 (socket says 172.21.15.113:44734) 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c2d9d700 1 -- 172.21.15.113:0/1404964574 learned_addr learned my addr 172.21.15.113:0/1404964574 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c259c700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa9c415b240 0x7fa9c41f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c359e700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9c415bb60 0x7fa9c41f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c2d9d700 1 -- 172.21.15.113:0/1404964574 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa9c415b240 msgr2=0x7fa9c41f5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c2d9d700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa9c415b240 0x7fa9c41f5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c2d9d700 1 -- 172.21.15.113:0/1404964574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9c415bb60 msgr2=0x7fa9c41f5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c2d9d700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9c415bb60 0x7fa9c41f5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c2d9d700 1 -- 172.21.15.113:0/1404964574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9bc009b30 con 0x7fa9c406c2e0 2024-01-15T11:25:45.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.283+0000 7fa9c2d9d700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9c406c2e0 0x7fa9c4154020 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa9b800c520 tx=0x7fa9b800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:45.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.284+0000 7fa9ab7fe700 1 -- 172.21.15.113:0/1404964574 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8005cb0 con 0x7fa9c406c2e0 2024-01-15T11:25:45.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.284+0000 7fa9ab7fe700 1 -- 172.21.15.113:0/1404964574 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b8005e10 con 0x7fa9c406c2e0 2024-01-15T11:25:45.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.284+0000 7fa9ab7fe700 1 -- 172.21.15.113:0/1404964574 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b80072d0 con 0x7fa9c406c2e0 2024-01-15T11:25:45.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.284+0000 7fa9c922a700 1 -- 172.21.15.113:0/1404964574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b40094a0 con 0x7fa9c406c2e0 2024-01-15T11:25:45.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.284+0000 7fa9c922a700 1 -- 172.21.15.113:0/1404964574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c40cfa30 con 0x7fa9c406c2e0 2024-01-15T11:25:45.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.285+0000 7fa9c922a700 1 -- 172.21.15.113:0/1404964574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c40d0990 con 0x7fa9c406c2e0 2024-01-15T11:25:45.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.285+0000 7fa9ab7fe700 1 -- 172.21.15.113:0/1404964574 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa9b8007430 con 0x7fa9c406c2e0 2024-01-15T11:25:45.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.285+0000 7fa9ab7fe700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa9b006b350 0x7fa9b006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.286+0000 7fa9ab7fe700 1 -- 172.21.15.113:0/1404964574 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9b8088fa0 con 0x7fa9c406c2e0 2024-01-15T11:25:45.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.286+0000 7fa9c259c700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa9b006b350 0x7fa9b006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.286+0000 7fa9c259c700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa9b006b350 0x7fa9b006d810 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4000c00 tx=0x7fa9b401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:45.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.290+0000 7fa9ab7fe700 1 -- 172.21.15.113:0/1404964574 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9b80bbe90 con 0x7fa9c406c2e0 2024-01-15T11:25:45.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.480+0000 7fa9c922a700 1 -- 172.21.15.113:0/1404964574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa9c4069020 con 0x7fa9c406c2e0 2024-01-15T11:25:45.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.480+0000 7fa9ab7fe700 1 -- 172.21.15.113:0/1404964574 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa9b8017020 con 0x7fa9c406c2e0 2024-01-15T11:25:45.479 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-15T11:25:45.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.482+0000 7fa9c922a700 1 -- 172.21.15.113:0/1404964574 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa9b006b350 msgr2=0x7fa9b006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.482+0000 7fa9c922a700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa9b006b350 0x7fa9b006d810 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4000c00 tx=0x7fa9b401c040 comp rx=0 tx=0).stop 2024-01-15T11:25:45.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.482+0000 7fa9c922a700 1 -- 172.21.15.113:0/1404964574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9c406c2e0 msgr2=0x7fa9c4154020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.482+0000 7fa9c922a700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9c406c2e0 0x7fa9c4154020 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa9b800c520 tx=0x7fa9b800c8e0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.482+0000 7fa9c922a700 1 -- 172.21.15.113:0/1404964574 shutdown_connections 2024-01-15T11:25:45.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.482+0000 7fa9c922a700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa9b006b350 0x7fa9b006d810 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.482+0000 7fa9c922a700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa9c415bb60 0x7fa9c41f5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.482+0000 7fa9c922a700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9c406c2e0 0x7fa9c4154020 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.482+0000 7fa9c922a700 1 --2- 172.21.15.113:0/1404964574 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa9c415b240 0x7fa9c41f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.482+0000 7fa9c922a700 1 -- 172.21.15.113:0/1404964574 >> 172.21.15.113:0/1404964574 conn(0x7fa9c40d0c00 msgr2=0x7fa9c4150e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:45.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.483+0000 7fa9c922a700 1 -- 172.21.15.113:0/1404964574 shutdown_connections 2024-01-15T11:25:45.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.483+0000 7fa9c922a700 1 -- 172.21.15.113:0/1404964574 wait complete. 2024-01-15T11:25:45.568 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) Test that when one clone is in progress, one progress bar is printed 2024-01-15T11:25:45.569 INFO:tasks.cephfs_test_runner:test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T11:25:45.569 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:45.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.691+0000 7f8c011fd700 1 Processor -- start 2024-01-15T11:25:45.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.691+0000 7f8c011fd700 1 -- start start 2024-01-15T11:25:45.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.692+0000 7f8c011fd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8bfc1473a0 0x7f8bfc15e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.692+0000 7f8c011fd700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8bfc147780 0x7f8bfc06c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.692+0000 7f8c011fd700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8bfc06dc20 0x7f8bfc073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.692+0000 7f8c011fd700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bfc06c9a0 con 0x7f8bfc1473a0 2024-01-15T11:25:45.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.692+0000 7f8c011fd700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bfc15eb00 con 0x7f8bfc147780 2024-01-15T11:25:45.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.692+0000 7f8c011fd700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc15ede0 con 0x7f8bfc06dc20 2024-01-15T11:25:45.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.692+0000 7f8bfad9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8bfc1473a0 0x7f8bfc15e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.692+0000 7f8bfad9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8bfc1473a0 0x7f8bfc15e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44750/0 (socket says 172.21.15.113:44750) 2024-01-15T11:25:45.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.692+0000 7f8bfad9d700 1 -- 172.21.15.113:0/2842821503 learned_addr learned my addr 172.21.15.113:0/2842821503 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:45.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.692+0000 7f8bfa59c700 1 --2- 172.21.15.113:0/2842821503 >> v2:172.21.15.141:3300/0 conn(0x7f8bfc147780 0x7f8bfc06c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.693+0000 7f8bfb59e700 1 --2- 172.21.15.113:0/2842821503 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8bfc06dc20 0x7f8bfc073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.693+0000 7f8bfad9d700 1 -- 172.21.15.113:0/2842821503 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8bfc06dc20 msgr2=0x7f8bfc073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.693+0000 7f8bfad9d700 1 --2- 172.21.15.113:0/2842821503 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8bfc06dc20 0x7f8bfc073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.693+0000 7f8bfad9d700 1 -- 172.21.15.113:0/2842821503 >> v2:172.21.15.141:3300/0 conn(0x7f8bfc147780 msgr2=0x7f8bfc06c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.693+0000 7f8bfad9d700 1 --2- 172.21.15.113:0/2842821503 >> v2:172.21.15.141:3300/0 conn(0x7f8bfc147780 0x7f8bfc06c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.693+0000 7f8bfad9d700 1 -- 172.21.15.113:0/2842821503 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc15f0c0 con 0x7f8bfc1473a0 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.693+0000 7f8bfad9d700 1 --2- 172.21.15.113:0/2842821503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bfc1473a0 0x7f8bfc15e4b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc147d30 tx=0x7f8be4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e4ac7d005e2f58 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.693+0000 7f8bf9d9b700 1 -- 172.21.15.113:0/2842821503 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4011830 con 0x7f8bfc1473a0 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.693+0000 7f8bf9d9b700 1 -- 172.21.15.113:0/2842821503 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be4011990 con 0x7f8bfc1473a0 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.693+0000 7f8bf9d9b700 1 -- 172.21.15.113:0/2842821503 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4011d20 con 0x7f8bfc1473a0 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.694+0000 7f8c011fd700 1 -- 172.21.15.113:0/2842821503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bfc1473a0 msgr2=0x7f8bfc15e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.694+0000 7f8c011fd700 1 --2- 172.21.15.113:0/2842821503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bfc1473a0 0x7f8bfc15e4b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc147d30 tx=0x7f8be4009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.694+0000 7f8c011fd700 1 -- 172.21.15.113:0/2842821503 shutdown_connections 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.694+0000 7f8c011fd700 1 --2- 172.21.15.113:0/2842821503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bfc1473a0 0x7f8bfc15e4b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.694+0000 7f8c011fd700 1 --2- 172.21.15.113:0/2842821503 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8bfc06dc20 0x7f8bfc073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.694+0000 7f8c011fd700 1 --2- 172.21.15.113:0/2842821503 >> v2:172.21.15.141:3300/0 conn(0x7f8bfc147780 0x7f8bfc06c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.694+0000 7f8c011fd700 1 -- 172.21.15.113:0/2842821503 >> 172.21.15.113:0/2842821503 conn(0x7f8bfc0b3130 msgr2=0x7f8bfc0d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.694+0000 7f8c011fd700 1 -- 172.21.15.113:0/2842821503 shutdown_connections 2024-01-15T11:25:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.694+0000 7f8c011fd700 1 -- 172.21.15.113:0/2842821503 wait complete. 2024-01-15T11:25:45.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.695+0000 7f8c011fd700 1 Processor -- start 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.695+0000 7f8c011fd700 1 -- start start 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8c011fd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bfc06dc20 0x7f8bfc143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8c011fd700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8bfc1473a0 0x7f8bfc1f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8c011fd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bfc147780 0x7f8bfc06b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8c011fd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0b1450 con 0x7f8bfc06dc20 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8c011fd700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0cd980 con 0x7f8bfc147780 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8c011fd700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0b0e90 con 0x7f8bfc1473a0 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8bfad9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bfc06dc20 0x7f8bfc143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8bfad9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bfc06dc20 0x7f8bfc143cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44766/0 (socket says 172.21.15.113:44766) 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8bfad9d700 1 -- 172.21.15.113:0/1326526792 learned_addr learned my addr 172.21.15.113:0/1326526792 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8bfb59e700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bfc147780 0x7f8bfc06b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8bfa59c700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8bfc1473a0 0x7f8bfc1f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8bfad9d700 1 -- 172.21.15.113:0/1326526792 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8bfc1473a0 msgr2=0x7f8bfc1f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8bfad9d700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8bfc1473a0 0x7f8bfc1f4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8bfad9d700 1 -- 172.21.15.113:0/1326526792 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bfc147780 msgr2=0x7f8bfc06b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8bfad9d700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bfc147780 0x7f8bfc06b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.696+0000 7f8bfad9d700 1 -- 172.21.15.113:0/1326526792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be4009550 con 0x7f8bfc06dc20 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.697+0000 7f8bfad9d700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bfc06dc20 0x7f8bfc143cb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8be4000c00 tx=0x7f8be4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.697+0000 7f8bf37fe700 1 -- 172.21.15.113:0/1326526792 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4004300 con 0x7f8bfc06dc20 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.697+0000 7f8bf37fe700 1 -- 172.21.15.113:0/1326526792 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be401f070 con 0x7f8bfc06dc20 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.697+0000 7f8c011fd700 1 -- 172.21.15.113:0/1326526792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bfc0b1170 con 0x7f8bfc06dc20 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.697+0000 7f8bf37fe700 1 -- 172.21.15.113:0/1326526792 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be40036a0 con 0x7f8bfc06dc20 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.697+0000 7f8c011fd700 1 -- 172.21.15.113:0/1326526792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bfc1441f0 con 0x7f8bfc06dc20 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.698+0000 7f8c011fd700 1 -- 172.21.15.113:0/1326526792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bfc0cd7e0 con 0x7f8bfc06dc20 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.698+0000 7f8bf37fe700 1 -- 172.21.15.113:0/1326526792 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8be4021030 con 0x7f8bfc06dc20 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.698+0000 7f8bf37fe700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8bdc06b350 0x7f8bdc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.699+0000 7f8bf37fe700 1 -- 172.21.15.113:0/1326526792 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8be408adc0 con 0x7f8bfc06dc20 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.699+0000 7f8bfa59c700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8bdc06b350 0x7f8bdc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:45.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.699+0000 7f8bfa59c700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8bdc06b350 0x7f8bdc06d810 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0b22c0 tx=0x7f8bec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:45.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.703+0000 7f8bf37fe700 1 -- 172.21.15.113:0/1326526792 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8bfc0cd7e0 con 0x7f8bfc06dc20 2024-01-15T11:25:45.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.894+0000 7f8c011fd700 1 -- 172.21.15.113:0/1326526792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bfc0cd7e0 con 0x7f8bfc06dc20 2024-01-15T11:25:45.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.895+0000 7f8bf37fe700 1 -- 172.21.15.113:0/1326526792 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f8bfc0cd7e0 con 0x7f8bfc06dc20 2024-01-15T11:25:45.894 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:45.894 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:45.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.897+0000 7f8c011fd700 1 -- 172.21.15.113:0/1326526792 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8bdc06b350 msgr2=0x7f8bdc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.897+0000 7f8c011fd700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8bdc06b350 0x7f8bdc06d810 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0b22c0 tx=0x7f8bec008040 comp rx=0 tx=0).stop 2024-01-15T11:25:45.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.897+0000 7f8c011fd700 1 -- 172.21.15.113:0/1326526792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bfc06dc20 msgr2=0x7f8bfc143cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:45.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.897+0000 7f8c011fd700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bfc06dc20 0x7f8bfc143cb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8be4000c00 tx=0x7f8be4004750 comp rx=0 tx=0).stop 2024-01-15T11:25:45.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.897+0000 7f8c011fd700 1 -- 172.21.15.113:0/1326526792 shutdown_connections 2024-01-15T11:25:45.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.897+0000 7f8c011fd700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8bdc06b350 0x7f8bdc06d810 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.897+0000 7f8c011fd700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8bfc147780 0x7f8bfc06b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.897+0000 7f8c011fd700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bfc06dc20 0x7f8bfc143cb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.897+0000 7f8c011fd700 1 --2- 172.21.15.113:0/1326526792 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8bfc1473a0 0x7f8bfc1f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:45.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.897+0000 7f8c011fd700 1 -- 172.21.15.113:0/1326526792 >> 172.21.15.113:0/1326526792 conn(0x7f8bfc0b3130 msgr2=0x7f8bfc146600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:45.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.897+0000 7f8c011fd700 1 -- 172.21.15.113:0/1326526792 shutdown_connections 2024-01-15T11:25:45.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:45.897+0000 7f8c011fd700 1 -- 172.21.15.113:0/1326526792 wait complete. 2024-01-15T11:25:45.908 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T11:25:13.398220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T11:25:14.589439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:45.909 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:46.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.034+0000 7fb68f304700 1 Processor -- start 2024-01-15T11:25:46.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.034+0000 7fb68f304700 1 -- start start 2024-01-15T11:25:46.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.035+0000 7fb68f304700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb68815b240 0x7fb68806e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.035+0000 7fb68f304700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb68806e9a0 0x7fb68815b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.035+0000 7fb68f304700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68806eeb0 con 0x7fb68815bb60 2024-01-15T11:25:46.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.035+0000 7fb68f304700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb68806f190 con 0x7fb68815b240 2024-01-15T11:25:46.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.035+0000 7fb68f304700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68806f470 con 0x7fb68806e9a0 2024-01-15T11:25:46.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.035+0000 7fb68d0a0700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb68815b240 0x7fb68806e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.035+0000 7fb68d8a1700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb68815bb60 0x7fb68806b2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42240/0 (socket says 172.21.15.113:42240) 2024-01-15T11:25:46.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.035+0000 7fb68d8a1700 1 -- 172.21.15.113:0/1758277636 learned_addr learned my addr 172.21.15.113:0/1758277636 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:46.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.035+0000 7fb68c89f700 1 --2- 172.21.15.113:0/1758277636 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb68806e9a0 0x7fb68815b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb68d0a0700 1 -- 172.21.15.113:0/1758277636 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb68806e9a0 msgr2=0x7fb68815b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb68d0a0700 1 --2- 172.21.15.113:0/1758277636 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb68806e9a0 0x7fb68815b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb68d0a0700 1 -- 172.21.15.113:0/1758277636 >> v1:172.21.15.113:6789/0 conn(0x7fb68815bb60 legacy=0x7fb68806b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb68d0a0700 1 -- 172.21.15.113:0/1758277636 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68806f750 con 0x7fb68815b240 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb67ffff700 1 -- 172.21.15.113:0/1758277636 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959314916 0 0) 0x7fb68806eeb0 con 0x7fb68815bb60 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb68c89f700 1 --2- 172.21.15.113:0/1758277636 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb68806e9a0 0x7fb68815b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb68d0a0700 1 --2- 172.21.15.113:0/1758277636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb68815b240 0x7fb68806e460 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb678009840 tx=0x7fb678009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e497779915ff99d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb67ffff700 1 -- 172.21.15.113:0/1758277636 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678011830 con 0x7fb68815b240 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb67ffff700 1 -- 172.21.15.113:0/1758277636 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb678011990 con 0x7fb68815b240 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb67ffff700 1 -- 172.21.15.113:0/1758277636 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678011d20 con 0x7fb68815b240 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb68f304700 1 -- 172.21.15.113:0/1758277636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb68815b240 msgr2=0x7fb68806e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb68f304700 1 --2- 172.21.15.113:0/1758277636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb68815b240 0x7fb68806e460 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb678009840 tx=0x7fb678009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb68f304700 1 -- 172.21.15.113:0/1758277636 shutdown_connections 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb68f304700 1 --2- 172.21.15.113:0/1758277636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb68815b240 0x7fb68806e460 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb68f304700 1 --2- 172.21.15.113:0/1758277636 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb68806e9a0 0x7fb68815b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.036+0000 7fb68f304700 1 -- 172.21.15.113:0/1758277636 >> 172.21.15.113:0/1758277636 conn(0x7fb6880d0c00 msgr2=0x7fb6880d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:46.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.037+0000 7fb68f304700 1 -- 172.21.15.113:0/1758277636 shutdown_connections 2024-01-15T11:25:46.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.037+0000 7fb68f304700 1 -- 172.21.15.113:0/1758277636 wait complete. 2024-01-15T11:25:46.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.038+0000 7fb68f304700 1 Processor -- start 2024-01-15T11:25:46.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.038+0000 7fb68f304700 1 -- start start 2024-01-15T11:25:46.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.038+0000 7fb68f304700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb68806e9a0 0x7fb68809aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.038+0000 7fb68f304700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb68815b240 0x7fb688144730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.038+0000 7fb68f304700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb68815bb60 0x7fb68809e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.038+0000 7fb68f304700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6881f22a0 con 0x7fb68815b240 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.038+0000 7fb68f304700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6881f1690 con 0x7fb68806e9a0 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.038+0000 7fb68f304700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6881f1970 con 0x7fb68815bb60 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.038+0000 7fb68c89f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb68815b240 0x7fb688144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68c89f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb68815b240 0x7fb688144730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44780/0 (socket says 172.21.15.113:44780) 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68c89f700 1 -- 172.21.15.113:0/875415883 learned_addr learned my addr 172.21.15.113:0/875415883 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68d0a0700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb68806e9a0 0x7fb68809aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68d8a1700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb68815bb60 0x7fb68809e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68c89f700 1 -- 172.21.15.113:0/875415883 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb68815bb60 msgr2=0x7fb68809e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68c89f700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb68815bb60 0x7fb68809e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68c89f700 1 -- 172.21.15.113:0/875415883 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb68806e9a0 msgr2=0x7fb68809aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68c89f700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb68806e9a0 0x7fb68809aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68c89f700 1 -- 172.21.15.113:0/875415883 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6780094a0 con 0x7fb68815b240 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68d0a0700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb68806e9a0 0x7fb68809aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68c89f700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb68815b240 0x7fb688144730 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c7f0 tx=0x7fb68400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb67dffb700 1 -- 172.21.15.113:0/875415883 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684011800 con 0x7fb68815b240 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb67dffb700 1 -- 172.21.15.113:0/875415883 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb684011960 con 0x7fb68815b240 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68f304700 1 -- 172.21.15.113:0/875415883 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6881f1c50 con 0x7fb68815b240 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb67dffb700 1 -- 172.21.15.113:0/875415883 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684011d10 con 0x7fb68815b240 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.039+0000 7fb68f304700 1 -- 172.21.15.113:0/875415883 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68809e680 con 0x7fb68815b240 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.040+0000 7fb68f304700 1 -- 172.21.15.113:0/875415883 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68809e8a0 con 0x7fb68815b240 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.041+0000 7fb67dffb700 1 -- 172.21.15.113:0/875415883 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb684018360 con 0x7fb68815b240 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.041+0000 7fb67dffb700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb66c06b350 0x7fb66c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.041+0000 7fb67dffb700 1 -- 172.21.15.113:0/875415883 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb684089b60 con 0x7fb68815b240 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.041+0000 7fb68d0a0700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb66c06b350 0x7fb66c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.041+0000 7fb68d0a0700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb66c06b350 0x7fb66c06d810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb6780090b0 tx=0x7fb678008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:46.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.046+0000 7fb67dffb700 1 -- 172.21.15.113:0/875415883 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb68809e8a0 con 0x7fb68815b240 2024-01-15T11:25:46.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.234+0000 7fb68f304700 1 -- 172.21.15.113:0/875415883 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb68809e8a0 con 0x7fb68815b240 2024-01-15T11:25:46.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.235+0000 7fb67dffb700 1 -- 172.21.15.113:0/875415883 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb68809e8a0 con 0x7fb68815b240 2024-01-15T11:25:46.233 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:46.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.237+0000 7fb68f304700 1 -- 172.21.15.113:0/875415883 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb66c06b350 msgr2=0x7fb66c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.237+0000 7fb68f304700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb66c06b350 0x7fb66c06d810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb6780090b0 tx=0x7fb678008030 comp rx=0 tx=0).stop 2024-01-15T11:25:46.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.237+0000 7fb68f304700 1 -- 172.21.15.113:0/875415883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb68815b240 msgr2=0x7fb688144730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.237+0000 7fb68f304700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb68815b240 0x7fb688144730 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c7f0 tx=0x7fb68400cbb0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.237+0000 7fb68f304700 1 -- 172.21.15.113:0/875415883 shutdown_connections 2024-01-15T11:25:46.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.237+0000 7fb68f304700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb66c06b350 0x7fb66c06d810 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.237+0000 7fb68f304700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb68815bb60 0x7fb68809e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.237+0000 7fb68f304700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb68806e9a0 0x7fb68809aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.237+0000 7fb68f304700 1 --2- 172.21.15.113:0/875415883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb68815b240 0x7fb688144730 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.237+0000 7fb68f304700 1 -- 172.21.15.113:0/875415883 >> 172.21.15.113:0/875415883 conn(0x7fb6880d0c00 msgr2=0x7fb6880cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:46.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.237+0000 7fb68f304700 1 -- 172.21.15.113:0/875415883 shutdown_connections 2024-01-15T11:25:46.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.237+0000 7fb68f304700 1 -- 172.21.15.113:0/875415883 wait complete. 2024-01-15T11:25:46.248 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T11:25:46.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.373+0000 7f6f75d59700 1 Processor -- start 2024-01-15T11:25:46.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.373+0000 7f6f75d59700 1 -- start start 2024-01-15T11:25:46.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.373+0000 7f6f75d59700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6f7015e0c0 0x7f6f700afb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.373+0000 7f6f75d59700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6f700746a0 0x7f6f700add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.373+0000 7f6f75d59700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7015e4a0 con 0x7f6f700b3bf0 2024-01-15T11:25:46.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.373+0000 7f6f75d59700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7006aff0 con 0x7f6f7015e0c0 2024-01-15T11:25:46.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.373+0000 7f6f75d59700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7006b2d0 con 0x7f6f700746a0 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6f7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6f7015e0c0 0x7f6f700afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6f7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6f7015e0c0 0x7f6f700afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44792/0 (socket says 172.21.15.113:44792) 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6f7fe700 1 -- 172.21.15.113:0/2416238533 learned_addr learned my addr 172.21.15.113:0/2416238533 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6ffff700 1 --2- 172.21.15.113:0/2416238533 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6f700746a0 0x7f6f700add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6f7fe700 1 -- 172.21.15.113:0/2416238533 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6f700746a0 msgr2=0x7f6f700add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6f7fe700 1 --2- 172.21.15.113:0/2416238533 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6f700746a0 0x7f6f700add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6f7fe700 1 -- 172.21.15.113:0/2416238533 >> v1:172.21.15.141:6789/0 conn(0x7f6f700b3bf0 legacy=0x7f6f70073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6f7fe700 1 -- 172.21.15.113:0/2416238533 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7006b5b0 con 0x7f6f7015e0c0 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6ffff700 1 --2- 172.21.15.113:0/2416238533 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6f700746a0 0x7f6f700add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6f7fe700 1 --2- 172.21.15.113:0/2416238533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f7015e0c0 0x7f6f700afb80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009920 tx=0x7f6f60009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e63c138d4b8737d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6e7fc700 1 -- 172.21.15.113:0/2416238533 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60004230 con 0x7f6f7015e0c0 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6e7fc700 1 -- 172.21.15.113:0/2416238533 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f60004390 con 0x7f6f7015e0c0 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.374+0000 7f6f6e7fc700 1 -- 172.21.15.113:0/2416238533 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60009e70 con 0x7f6f7015e0c0 2024-01-15T11:25:46.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.375+0000 7f6f75d59700 1 -- 172.21.15.113:0/2416238533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f7015e0c0 msgr2=0x7f6f700afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.375+0000 7f6f75d59700 1 --2- 172.21.15.113:0/2416238533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f7015e0c0 0x7f6f700afb80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009920 tx=0x7f6f60009c30 comp rx=0 tx=0).stop 2024-01-15T11:25:46.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.375+0000 7f6f75d59700 1 -- 172.21.15.113:0/2416238533 shutdown_connections 2024-01-15T11:25:46.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.375+0000 7f6f75d59700 1 --2- 172.21.15.113:0/2416238533 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6f700746a0 0x7f6f700add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.375+0000 7f6f75d59700 1 --2- 172.21.15.113:0/2416238533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f7015e0c0 0x7f6f700afb80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.375+0000 7f6f75d59700 1 -- 172.21.15.113:0/2416238533 >> 172.21.15.113:0/2416238533 conn(0x7f6f700d0c00 msgr2=0x7f6f700d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:46.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.375+0000 7f6f75d59700 1 -- 172.21.15.113:0/2416238533 shutdown_connections 2024-01-15T11:25:46.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.375+0000 7f6f75d59700 1 -- 172.21.15.113:0/2416238533 wait complete. 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.376+0000 7f6f75d59700 1 Processor -- start 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.376+0000 7f6f75d59700 1 -- start start 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f75d59700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f700746a0 0x7f6f7013e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f75d59700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6f700b3bf0 0x7f6f70140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f75d59700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f7015e0c0 0x7f6f70140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f75d59700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70140290 con 0x7f6f7015e0c0 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f75d59700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7013e6e0 con 0x7f6f700746a0 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f75d59700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7013f680 con 0x7f6f700b3bf0 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f7015e0c0 0x7f6f70140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f7015e0c0 0x7f6f70140f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44798/0 (socket says 172.21.15.113:44798) 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6ffff700 1 -- 172.21.15.113:0/2063094968 learned_addr learned my addr 172.21.15.113:0/2063094968 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6effd700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6f700b3bf0 0x7f6f70140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6f7fe700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f700746a0 0x7f6f7013e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6ffff700 1 -- 172.21.15.113:0/2063094968 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6f700b3bf0 msgr2=0x7f6f70140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6ffff700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6f700b3bf0 0x7f6f70140a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6ffff700 1 -- 172.21.15.113:0/2063094968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f700746a0 msgr2=0x7f6f7013e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6ffff700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f700746a0 0x7f6f7013e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6ffff700 1 -- 172.21.15.113:0/2063094968 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f60009550 con 0x7f6f7015e0c0 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6effd700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6f700b3bf0 0x7f6f70140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6f7fe700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f700746a0 0x7f6f7013e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f6ffff700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f7015e0c0 0x7f6f70140f90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800cef0 tx=0x7f6f68007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f57fff700 1 -- 172.21.15.113:0/2063094968 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6800ae20 con 0x7f6f7015e0c0 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f57fff700 1 -- 172.21.15.113:0/2063094968 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f68015440 con 0x7f6f7015e0c0 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f57fff700 1 -- 172.21.15.113:0/2063094968 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f680157f0 con 0x7f6f7015e0c0 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.377+0000 7f6f75d59700 1 -- 172.21.15.113:0/2063094968 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7013f960 con 0x7f6f7015e0c0 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.378+0000 7f6f75d59700 1 -- 172.21.15.113:0/2063094968 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7013ff20 con 0x7f6f7015e0c0 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.379+0000 7f6f75d59700 1 -- 172.21.15.113:0/2063094968 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f700b4060 con 0x7f6f7015e0c0 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.379+0000 7f6f57fff700 1 -- 172.21.15.113:0/2063094968 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6f6800d430 con 0x7f6f7015e0c0 2024-01-15T11:25:46.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.379+0000 7f6f57fff700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6f5c06b350 0x7f6f5c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.379+0000 7f6f57fff700 1 -- 172.21.15.113:0/2063094968 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f6808a160 con 0x7f6f7015e0c0 2024-01-15T11:25:46.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.379+0000 7f6f6f7fe700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6f5c06b350 0x7f6f5c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.380+0000 7f6f6f7fe700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6f5c06b350 0x7f6f5c06d810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009130 tx=0x7f6f60008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:46.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.384+0000 7f6f57fff700 1 -- 172.21.15.113:0/2063094968 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f700b4060 con 0x7f6f7015e0c0 2024-01-15T11:25:46.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.574+0000 7f6f75d59700 1 -- 172.21.15.113:0/2063094968 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6f700b4060 con 0x7f6f7015e0c0 2024-01-15T11:25:46.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.574+0000 7f6f57fff700 1 -- 172.21.15.113:0/2063094968 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6f700b4060 con 0x7f6f7015e0c0 2024-01-15T11:25:46.573 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-15T11:25:46.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.576+0000 7f6f75d59700 1 -- 172.21.15.113:0/2063094968 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6f5c06b350 msgr2=0x7f6f5c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.576+0000 7f6f75d59700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6f5c06b350 0x7f6f5c06d810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009130 tx=0x7f6f60008030 comp rx=0 tx=0).stop 2024-01-15T11:25:46.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.576+0000 7f6f75d59700 1 -- 172.21.15.113:0/2063094968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f7015e0c0 msgr2=0x7f6f70140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.576+0000 7f6f75d59700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f7015e0c0 0x7f6f70140f90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800cef0 tx=0x7f6f68007f60 comp rx=0 tx=0).stop 2024-01-15T11:25:46.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.576+0000 7f6f75d59700 1 -- 172.21.15.113:0/2063094968 shutdown_connections 2024-01-15T11:25:46.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.576+0000 7f6f75d59700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6f5c06b350 0x7f6f5c06d810 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.576+0000 7f6f75d59700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f7015e0c0 0x7f6f70140f90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.576+0000 7f6f75d59700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f700746a0 0x7f6f7013e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.576+0000 7f6f75d59700 1 --2- 172.21.15.113:0/2063094968 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6f700b3bf0 0x7f6f70140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.576+0000 7f6f75d59700 1 -- 172.21.15.113:0/2063094968 >> 172.21.15.113:0/2063094968 conn(0x7f6f700d0c00 msgr2=0x7f6f7006f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:46.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.577+0000 7f6f75d59700 1 -- 172.21.15.113:0/2063094968 shutdown_connections 2024-01-15T11:25:46.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.577+0000 7f6f75d59700 1 -- 172.21.15.113:0/2063094968 wait complete. 2024-01-15T11:25:46.587 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T11:25:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.712+0000 7f654b154700 1 Processor -- start 2024-01-15T11:25:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.712+0000 7f654b154700 1 -- start start 2024-01-15T11:25:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.713+0000 7f654b154700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6544161690 0x7f65440d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.713+0000 7f654b154700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f654409b110 0x7f65440d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.713+0000 7f654b154700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65440d0a70 con 0x7f6544161a70 2024-01-15T11:25:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.713+0000 7f654b154700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65440d1c10 con 0x7f6544161690 2024-01-15T11:25:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.713+0000 7f654b154700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65440b2ab0 con 0x7f654409b110 2024-01-15T11:25:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.713+0000 7f6548ef0700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6544161690 0x7f65440d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.713+0000 7f65496f1700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f654409b110 0x7f65440d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.713+0000 7f6543fff700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f6544161a70 0x7f654409aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:55140/0 (socket says 172.21.15.113:55140) 2024-01-15T11:25:46.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.713+0000 7f6543fff700 1 -- 172.21.15.113:0/3033101199 learned_addr learned my addr 172.21.15.113:0/3033101199 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:46.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.714+0000 7f65496f1700 1 -- 172.21.15.113:0/3033101199 >> v2:172.21.15.141:3300/0 conn(0x7f6544161690 msgr2=0x7f65440d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.714+0000 7f65496f1700 1 --2- 172.21.15.113:0/3033101199 >> v2:172.21.15.141:3300/0 conn(0x7f6544161690 0x7f65440d0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.714+0000 7f65496f1700 1 -- 172.21.15.113:0/3033101199 >> v1:172.21.15.141:6789/0 conn(0x7f6544161a70 legacy=0x7f654409aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.714+0000 7f65496f1700 1 -- 172.21.15.113:0/3033101199 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65440b2d90 con 0x7f654409b110 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.714+0000 7f6548ef0700 1 --2- 172.21.15.113:0/3033101199 >> v2:172.21.15.141:3300/0 conn(0x7f6544161690 0x7f65440d0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.714+0000 7f65437fe700 1 -- 172.21.15.113:0/3033101199 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232272419 0 0) 0x7f65440d0a70 con 0x7f6544161a70 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.714+0000 7f65496f1700 1 --2- 172.21.15.113:0/3033101199 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f654409b110 0x7f65440d2970 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f653c009d90 tx=0x7f653c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9485a2105091ae29 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.714+0000 7f65437fe700 1 -- 172.21.15.113:0/3033101199 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c00e070 con 0x7f654409b110 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.714+0000 7f65437fe700 1 -- 172.21.15.113:0/3033101199 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653c00bb20 con 0x7f654409b110 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.714+0000 7f65437fe700 1 -- 172.21.15.113:0/3033101199 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c012430 con 0x7f654409b110 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.715+0000 7f654b154700 1 -- 172.21.15.113:0/3033101199 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f654409b110 msgr2=0x7f65440d2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.715+0000 7f654b154700 1 --2- 172.21.15.113:0/3033101199 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f654409b110 0x7f65440d2970 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f653c009d90 tx=0x7f653c00b330 comp rx=0 tx=0).stop 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.715+0000 7f654b154700 1 -- 172.21.15.113:0/3033101199 shutdown_connections 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.715+0000 7f654b154700 1 --2- 172.21.15.113:0/3033101199 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f654409b110 0x7f65440d2970 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.715+0000 7f654b154700 1 --2- 172.21.15.113:0/3033101199 >> v2:172.21.15.141:3300/0 conn(0x7f6544161690 0x7f65440d0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.715+0000 7f654b154700 1 -- 172.21.15.113:0/3033101199 >> 172.21.15.113:0/3033101199 conn(0x7f65440d1e50 msgr2=0x7f65440d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:46.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.715+0000 7f654b154700 1 -- 172.21.15.113:0/3033101199 shutdown_connections 2024-01-15T11:25:46.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.715+0000 7f654b154700 1 -- 172.21.15.113:0/3033101199 wait complete. 2024-01-15T11:25:46.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.716+0000 7f654b154700 1 Processor -- start 2024-01-15T11:25:46.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.716+0000 7f654b154700 1 -- start start 2024-01-15T11:25:46.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f654b154700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f654409b110 0x7f654415d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f654b154700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6544161690 0x7f65440c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f654b154700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544161a70 0x7f6544070020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f654b154700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65440b4d90 con 0x7f6544161a70 2024-01-15T11:25:46.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f654b154700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65440c79e0 con 0x7f654409b110 2024-01-15T11:25:46.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f654b154700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65440c7160 con 0x7f6544161690 2024-01-15T11:25:46.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f65496f1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544161a70 0x7f6544070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f65496f1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544161a70 0x7f6544070020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44812/0 (socket says 172.21.15.113:44812) 2024-01-15T11:25:46.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f65496f1700 1 -- 172.21.15.113:0/4261609174 learned_addr learned my addr 172.21.15.113:0/4261609174 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:46.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f6548ef0700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f654409b110 0x7f654415d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f65496f1700 1 -- 172.21.15.113:0/4261609174 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6544161690 msgr2=0x7f65440c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f65496f1700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6544161690 0x7f65440c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f65496f1700 1 -- 172.21.15.113:0/4261609174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f654409b110 msgr2=0x7f654415d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f65496f1700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f654409b110 0x7f654415d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.717+0000 7f65496f1700 1 -- 172.21.15.113:0/4261609174 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65380094a0 con 0x7f6544161a70 2024-01-15T11:25:46.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.718+0000 7f65496f1700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544161a70 0x7f6544070020 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f653c013000 tx=0x7f653c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:46.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.718+0000 7f65417fa700 1 -- 172.21.15.113:0/4261609174 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c012430 con 0x7f6544161a70 2024-01-15T11:25:46.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.718+0000 7f65417fa700 1 -- 172.21.15.113:0/4261609174 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653c020070 con 0x7f6544161a70 2024-01-15T11:25:46.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.718+0000 7f654b154700 1 -- 172.21.15.113:0/4261609174 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653c009a70 con 0x7f6544161a70 2024-01-15T11:25:46.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.718+0000 7f65417fa700 1 -- 172.21.15.113:0/4261609174 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c004280 con 0x7f6544161a70 2024-01-15T11:25:46.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.718+0000 7f654b154700 1 -- 172.21.15.113:0/4261609174 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65440c7700 con 0x7f6544161a70 2024-01-15T11:25:46.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.719+0000 7f654b154700 1 -- 172.21.15.113:0/4261609174 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65440b4af0 con 0x7f6544161a70 2024-01-15T11:25:46.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.719+0000 7f65417fa700 1 -- 172.21.15.113:0/4261609174 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f653c0043e0 con 0x7f6544161a70 2024-01-15T11:25:46.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.719+0000 7f65417fa700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f652806b350 0x7f652806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:46.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.720+0000 7f65417fa700 1 -- 172.21.15.113:0/4261609174 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f653c022070 con 0x7f6544161a70 2024-01-15T11:25:46.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.720+0000 7f6548ef0700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f652806b350 0x7f652806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:46.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.720+0000 7f6548ef0700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f652806b350 0x7f652806d810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6538009d00 tx=0x7f6538008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:46.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.724+0000 7f65417fa700 1 -- 172.21.15.113:0/4261609174 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65440b4af0 con 0x7f6544161a70 2024-01-15T11:25:46.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.913+0000 7f654b154700 1 -- 172.21.15.113:0/4261609174 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f65440d1710 con 0x7f6544161a70 2024-01-15T11:25:46.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.914+0000 7f65417fa700 1 -- 172.21.15.113:0/4261609174 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f65440d1710 con 0x7f6544161a70 2024-01-15T11:25:46.912 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-15T11:25:46.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.915+0000 7f654b154700 1 -- 172.21.15.113:0/4261609174 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f652806b350 msgr2=0x7f652806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.915+0000 7f654b154700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f652806b350 0x7f652806d810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6538009d00 tx=0x7f6538008040 comp rx=0 tx=0).stop 2024-01-15T11:25:46.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.915+0000 7f654b154700 1 -- 172.21.15.113:0/4261609174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544161a70 msgr2=0x7f6544070020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:46.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.915+0000 7f654b154700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544161a70 0x7f6544070020 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f653c013000 tx=0x7f653c004ad0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.916+0000 7f654b154700 1 -- 172.21.15.113:0/4261609174 shutdown_connections 2024-01-15T11:25:46.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.916+0000 7f654b154700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f652806b350 0x7f652806d810 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.916+0000 7f654b154700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544161a70 0x7f6544070020 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.916+0000 7f654b154700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f654409b110 0x7f654415d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.916+0000 7f654b154700 1 --2- 172.21.15.113:0/4261609174 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6544161690 0x7f65440c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:46.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.916+0000 7f654b154700 1 -- 172.21.15.113:0/4261609174 >> 172.21.15.113:0/4261609174 conn(0x7f65440d1e50 msgr2=0x7f6544160270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:46.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.916+0000 7f654b154700 1 -- 172.21.15.113:0/4261609174 shutdown_connections 2024-01-15T11:25:46.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:46.916+0000 7f654b154700 1 -- 172.21.15.113:0/4261609174 wait complete. 2024-01-15T11:25:46.926 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-15T11:25:47.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.049+0000 7f1df9564700 1 Processor -- start 2024-01-15T11:25:47.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.049+0000 7f1df9564700 1 -- start start 2024-01-15T11:25:47.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.050+0000 7f1df9564700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1df415e1e0 0x7f1df40d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:47.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.050+0000 7f1df9564700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1df40d3a00 0x7f1df40d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:47.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.050+0000 7f1df9564700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df40d20b0 con 0x7f1df40d0db0 2024-01-15T11:25:47.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.050+0000 7f1df9564700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df40d3150 con 0x7f1df40d3a00 2024-01-15T11:25:47.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.050+0000 7f1df9564700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df40d1240 con 0x7f1df415e1e0 2024-01-15T11:25:47.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.050+0000 7f1df27fc700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1df40d3a00 0x7f1df40d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:47.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.050+0000 7f1df37fe700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f1df40d0db0 0x7f1df406aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:55148/0 (socket says 172.21.15.113:55148) 2024-01-15T11:25:47.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.050+0000 7f1df2ffd700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1df415e1e0 0x7f1df40d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:47.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.050+0000 7f1df37fe700 1 -- 172.21.15.113:0/4030401025 learned_addr learned my addr 172.21.15.113:0/4030401025 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:47.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.051+0000 7f1df27fc700 1 -- 172.21.15.113:0/4030401025 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1df415e1e0 msgr2=0x7f1df40d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:47.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.051+0000 7f1df27fc700 1 --2- 172.21.15.113:0/4030401025 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1df415e1e0 0x7f1df40d34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:47.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.051+0000 7f1df27fc700 1 -- 172.21.15.113:0/4030401025 >> v1:172.21.15.141:6789/0 conn(0x7f1df40d0db0 legacy=0x7f1df406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.051+0000 7f1df27fc700 1 -- 172.21.15.113:0/4030401025 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df40d2390 con 0x7f1df40d3a00 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.051+0000 7f1df1ffb700 1 -- 172.21.15.113:0/4030401025 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100272464 0 0) 0x7f1df40d20b0 con 0x7f1df40d0db0 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.051+0000 7f1df27fc700 1 --2- 172.21.15.113:0/4030401025 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1df40d3a00 0x7f1df40d0870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009840 tx=0x7f1de4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c68c5c1db7b9152d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.051+0000 7f1df1ffb700 1 -- 172.21.15.113:0/4030401025 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4011830 con 0x7f1df40d3a00 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.051+0000 7f1df1ffb700 1 -- 172.21.15.113:0/4030401025 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de4011990 con 0x7f1df40d3a00 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.051+0000 7f1df1ffb700 1 -- 172.21.15.113:0/4030401025 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4011d20 con 0x7f1df40d3a00 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.052+0000 7f1df9564700 1 -- 172.21.15.113:0/4030401025 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1df40d3a00 msgr2=0x7f1df40d0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.052+0000 7f1df9564700 1 --2- 172.21.15.113:0/4030401025 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1df40d3a00 0x7f1df40d0870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009840 tx=0x7f1de4009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.052+0000 7f1df9564700 1 -- 172.21.15.113:0/4030401025 shutdown_connections 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.052+0000 7f1df9564700 1 --2- 172.21.15.113:0/4030401025 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1df415e1e0 0x7f1df40d34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.052+0000 7f1df9564700 1 --2- 172.21.15.113:0/4030401025 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1df40d3a00 0x7f1df40d0870 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.052+0000 7f1df9564700 1 -- 172.21.15.113:0/4030401025 >> 172.21.15.113:0/4030401025 conn(0x7f1df40b5b50 msgr2=0x7f1df40b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.052+0000 7f1df9564700 1 -- 172.21.15.113:0/4030401025 shutdown_connections 2024-01-15T11:25:47.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.052+0000 7f1df9564700 1 -- 172.21.15.113:0/4030401025 wait complete. 2024-01-15T11:25:47.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.053+0000 7f1df9564700 1 Processor -- start 2024-01-15T11:25:47.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.053+0000 7f1df9564700 1 -- start start 2024-01-15T11:25:47.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df9564700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1df40d0db0 0x7f1df4148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:47.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df9564700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1df40d3a00 0x7f1df415e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:47.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df9564700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1df415e1e0 0x7f1df415ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:47.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df9564700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df415f2b0 con 0x7f1df415e1e0 2024-01-15T11:25:47.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df9564700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df415f590 con 0x7f1df40d3a00 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df9564700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df415f870 con 0x7f1df40d0db0 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1df415e1e0 0x7f1df415ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1df415e1e0 0x7f1df415ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44824/0 (socket says 172.21.15.113:44824) 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df37fe700 1 -- 172.21.15.113:0/3728541301 learned_addr learned my addr 172.21.15.113:0/3728541301 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df27fc700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1df40d3a00 0x7f1df415e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df2ffd700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1df40d0db0 0x7f1df4148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df37fe700 1 -- 172.21.15.113:0/3728541301 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1df40d0db0 msgr2=0x7f1df4148340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df37fe700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1df40d0db0 0x7f1df4148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df37fe700 1 -- 172.21.15.113:0/3728541301 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1df40d3a00 msgr2=0x7f1df415e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df37fe700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1df40d3a00 0x7f1df415e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df37fe700 1 -- 172.21.15.113:0/3728541301 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de40094a0 con 0x7f1df415e1e0 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1df37fe700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1df415e1e0 0x7f1df415ec80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00cca0 tx=0x7f1dec00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.054+0000 7f1ddb7fe700 1 -- 172.21.15.113:0/3728541301 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec00cea0 con 0x7f1df415e1e0 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.055+0000 7f1ddb7fe700 1 -- 172.21.15.113:0/3728541301 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dec00ade0 con 0x7f1df415e1e0 2024-01-15T11:25:47.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.055+0000 7f1df9564700 1 -- 172.21.15.113:0/3728541301 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df415fb50 con 0x7f1df415e1e0 2024-01-15T11:25:47.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.055+0000 7f1ddb7fe700 1 -- 172.21.15.113:0/3728541301 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec007310 con 0x7f1df415e1e0 2024-01-15T11:25:47.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.055+0000 7f1df9564700 1 -- 172.21.15.113:0/3728541301 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df4151370 con 0x7f1df415e1e0 2024-01-15T11:25:47.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.055+0000 7f1df9564700 1 -- 172.21.15.113:0/3728541301 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df4160070 con 0x7f1df415e1e0 2024-01-15T11:25:47.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.056+0000 7f1ddb7fe700 1 -- 172.21.15.113:0/3728541301 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1dec007470 con 0x7f1df415e1e0 2024-01-15T11:25:47.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.056+0000 7f1ddb7fe700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f1de006b350 0x7f1de006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:47.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.056+0000 7f1ddb7fe700 1 -- 172.21.15.113:0/3728541301 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1dec089a40 con 0x7f1df415e1e0 2024-01-15T11:25:47.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.056+0000 7f1df2ffd700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f1de006b350 0x7f1de006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:47.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.057+0000 7f1df2ffd700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f1de006b350 0x7f1de006d810 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009a10 tx=0x7f1de8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:47.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.061+0000 7f1ddb7fe700 1 -- 172.21.15.113:0/3728541301 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1dec08be60 con 0x7f1df415e1e0 2024-01-15T11:25:47.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:47.335+0000 7f1df9564700 1 -- 172.21.15.113:0/3728541301 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7f1df40751e0 con 0x7f1df415e1e0 2024-01-15T11:25:48.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.227+0000 7f1ddb7fe700 1 -- 172.21.15.113:0/3728541301 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v52) v1 ==== 155+0+0 (secure 0 0 0) 0x7f1dec018020 con 0x7f1df415e1e0 2024-01-15T11:25:48.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.231+0000 7f1df9564700 1 -- 172.21.15.113:0/3728541301 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f1de006b350 msgr2=0x7f1de006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:48.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.231+0000 7f1df9564700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f1de006b350 0x7f1de006d810 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009a10 tx=0x7f1de8008040 comp rx=0 tx=0).stop 2024-01-15T11:25:48.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.231+0000 7f1df9564700 1 -- 172.21.15.113:0/3728541301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1df415e1e0 msgr2=0x7f1df415ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:48.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.231+0000 7f1df9564700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1df415e1e0 0x7f1df415ec80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00cca0 tx=0x7f1dec00a340 comp rx=0 tx=0).stop 2024-01-15T11:25:48.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.231+0000 7f1df9564700 1 -- 172.21.15.113:0/3728541301 shutdown_connections 2024-01-15T11:25:48.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.231+0000 7f1df9564700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f1de006b350 0x7f1de006d810 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.231+0000 7f1df9564700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1df415e1e0 0x7f1df415ec80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.231+0000 7f1df9564700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1df40d0db0 0x7f1df4148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.231+0000 7f1df9564700 1 --2- 172.21.15.113:0/3728541301 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1df40d3a00 0x7f1df415e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.231+0000 7f1df9564700 1 -- 172.21.15.113:0/3728541301 >> 172.21.15.113:0/3728541301 conn(0x7f1df40b5b50 msgr2=0x7f1df40b4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:48.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.231+0000 7f1df9564700 1 -- 172.21.15.113:0/3728541301 shutdown_connections 2024-01-15T11:25:48.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.231+0000 7f1df9564700 1 -- 172.21.15.113:0/3728541301 wait complete. 2024-01-15T11:25:48.246 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T11:25:48.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.381+0000 7f81e5f82700 1 Processor -- start 2024-01-15T11:25:48.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.381+0000 7f81e5f82700 1 -- start start 2024-01-15T11:25:48.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81e5f82700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f81e00cffc0 0x7f81e00b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:48.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81e5f82700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f81e00d03a0 0x7f81e015f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:48.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81e5f82700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00b3860 con 0x7f81e015f700 2024-01-15T11:25:48.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81e5f82700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e015fb90 con 0x7f81e00cffc0 2024-01-15T11:25:48.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81e5f82700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e00b2520 con 0x7f81e00d03a0 2024-01-15T11:25:48.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81df7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f81e00cffc0 0x7f81e00b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:48.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81dffff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f81e015f700 0x7f81e009aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42246/0 (socket says 172.21.15.113:42246) 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81dffff700 1 -- 172.21.15.113:0/2039116212 learned_addr learned my addr 172.21.15.113:0/2039116212 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81df7fe700 1 -- 172.21.15.113:0/2039116212 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f81e00d03a0 msgr2=0x7f81e015f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81df7fe700 1 --2- 172.21.15.113:0/2039116212 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f81e00d03a0 0x7f81e015f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81df7fe700 1 -- 172.21.15.113:0/2039116212 >> v1:172.21.15.113:6789/0 conn(0x7f81e015f700 legacy=0x7f81e009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81df7fe700 1 -- 172.21.15.113:0/2039116212 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e00b2800 con 0x7f81e00cffc0 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81de7fc700 1 -- 172.21.15.113:0/2039116212 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482591896 0 0) 0x7f81d8001490 con 0x7f81e015f700 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81df7fe700 1 --2- 172.21.15.113:0/2039116212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e00cffc0 0x7f81e00b3320 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f81d4009840 tx=0x7f81d4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90117d85984bf869 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81de7fc700 1 -- 172.21.15.113:0/2039116212 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4011830 con 0x7f81e00cffc0 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81de7fc700 1 -- 172.21.15.113:0/2039116212 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d4011990 con 0x7f81e00cffc0 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.382+0000 7f81de7fc700 1 -- 172.21.15.113:0/2039116212 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4011d20 con 0x7f81e00cffc0 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.383+0000 7f81e5f82700 1 -- 172.21.15.113:0/2039116212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e00cffc0 msgr2=0x7f81e00b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.383+0000 7f81e5f82700 1 --2- 172.21.15.113:0/2039116212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e00cffc0 0x7f81e00b3320 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f81d4009840 tx=0x7f81d4009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.383+0000 7f81e5f82700 1 -- 172.21.15.113:0/2039116212 shutdown_connections 2024-01-15T11:25:48.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.383+0000 7f81e5f82700 1 --2- 172.21.15.113:0/2039116212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e00cffc0 0x7f81e00b3320 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.383+0000 7f81e5f82700 1 --2- 172.21.15.113:0/2039116212 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f81e00d03a0 0x7f81e015f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.383+0000 7f81e5f82700 1 -- 172.21.15.113:0/2039116212 >> 172.21.15.113:0/2039116212 conn(0x7f81e00d1e50 msgr2=0x7f81e00d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:48.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.383+0000 7f81e5f82700 1 -- 172.21.15.113:0/2039116212 shutdown_connections 2024-01-15T11:25:48.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.383+0000 7f81e5f82700 1 -- 172.21.15.113:0/2039116212 wait complete. 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.384+0000 7f81e5f82700 1 Processor -- start 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.384+0000 7f81e5f82700 1 -- start start 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81e5f82700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81e00cffc0 0x7f81e00cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81e5f82700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f81e00d03a0 0x7f81e0154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81e5f82700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e015f700 0x7f81e0150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81e5f82700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e00b7890 con 0x7f81e015f700 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81e5f82700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e00cbd70 con 0x7f81e00cffc0 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81e5f82700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e00b6f60 con 0x7f81e00d03a0 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81dffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e015f700 0x7f81e0150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81dffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e015f700 0x7f81e0150430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44852/0 (socket says 172.21.15.113:44852) 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81dffff700 1 -- 172.21.15.113:0/1681267472 learned_addr learned my addr 172.21.15.113:0/1681267472 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81df7fe700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81e00cffc0 0x7f81e00cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81deffd700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f81e00d03a0 0x7f81e0154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:48.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81dffff700 1 -- 172.21.15.113:0/1681267472 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f81e00d03a0 msgr2=0x7f81e0154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81dffff700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f81e00d03a0 0x7f81e0154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81dffff700 1 -- 172.21.15.113:0/1681267472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81e00cffc0 msgr2=0x7f81e00cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81dffff700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81e00cffc0 0x7f81e00cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81dffff700 1 -- 172.21.15.113:0/1681267472 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d40094a0 con 0x7f81e015f700 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.385+0000 7f81df7fe700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81e00cffc0 0x7f81e00cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.386+0000 7f81dffff700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e015f700 0x7f81e0150430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f81d800ccd0 tx=0x7f81d800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.386+0000 7f81c7fff700 1 -- 172.21.15.113:0/1681267472 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d800ad40 con 0x7f81e015f700 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.386+0000 7f81c7fff700 1 -- 172.21.15.113:0/1681267472 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d80070d0 con 0x7f81e015f700 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.386+0000 7f81e5f82700 1 -- 172.21.15.113:0/1681267472 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e00b7240 con 0x7f81e015f700 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.386+0000 7f81c7fff700 1 -- 172.21.15.113:0/1681267472 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8007480 con 0x7f81e015f700 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.386+0000 7f81e5f82700 1 -- 172.21.15.113:0/1681267472 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e0150970 con 0x7f81e015f700 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.387+0000 7f81e5f82700 1 -- 172.21.15.113:0/1681267472 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e00b7a40 con 0x7f81e015f700 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.387+0000 7f81c7fff700 1 -- 172.21.15.113:0/1681267472 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f81d80075e0 con 0x7f81e015f700 2024-01-15T11:25:48.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.388+0000 7f81c7fff700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f81cc06b350 0x7f81cc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:48.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.388+0000 7f81c7fff700 1 -- 172.21.15.113:0/1681267472 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81d80898c0 con 0x7f81e015f700 2024-01-15T11:25:48.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.388+0000 7f81df7fe700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f81cc06b350 0x7f81cc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:48.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.388+0000 7f81df7fe700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f81cc06b350 0x7f81cc06d810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f81d40090b0 tx=0x7f81d401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:48.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.392+0000 7f81c7fff700 1 -- 172.21.15.113:0/1681267472 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f81e00b7a40 con 0x7f81e015f700 2024-01-15T11:25:48.581 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.583+0000 7f81e5f82700 1 -- 172.21.15.113:0/1681267472 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f81e00b3860 con 0x7f81e015f700 2024-01-15T11:25:48.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.586+0000 7f81c7fff700 1 -- 172.21.15.113:0/1681267472 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f81e00b3860 con 0x7f81e015f700 2024-01-15T11:25:48.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.588+0000 7f81e5f82700 1 -- 172.21.15.113:0/1681267472 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f81cc06b350 msgr2=0x7f81cc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:48.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.588+0000 7f81e5f82700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f81cc06b350 0x7f81cc06d810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f81d40090b0 tx=0x7f81d401c040 comp rx=0 tx=0).stop 2024-01-15T11:25:48.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.588+0000 7f81e5f82700 1 -- 172.21.15.113:0/1681267472 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e015f700 msgr2=0x7f81e0150430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:48.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.588+0000 7f81e5f82700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e015f700 0x7f81e0150430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f81d800ccd0 tx=0x7f81d800a340 comp rx=0 tx=0).stop 2024-01-15T11:25:48.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.588+0000 7f81e5f82700 1 -- 172.21.15.113:0/1681267472 shutdown_connections 2024-01-15T11:25:48.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.588+0000 7f81e5f82700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f81cc06b350 0x7f81cc06d810 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.588+0000 7f81e5f82700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e015f700 0x7f81e0150430 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.588+0000 7f81e5f82700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81e00cffc0 0x7f81e00cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.588+0000 7f81e5f82700 1 --2- 172.21.15.113:0/1681267472 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f81e00d03a0 0x7f81e0154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.588+0000 7f81e5f82700 1 -- 172.21.15.113:0/1681267472 >> 172.21.15.113:0/1681267472 conn(0x7f81e00d1e50 msgr2=0x7f81e006efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:48.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.588+0000 7f81e5f82700 1 -- 172.21.15.113:0/1681267472 shutdown_connections 2024-01-15T11:25:48.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.588+0000 7f81e5f82700 1 -- 172.21.15.113:0/1681267472 wait complete. 2024-01-15T11:25:48.599 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T11:25:48.599 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T11:25:48.640 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T11:25:48.686 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T11:25:48.686 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:25:48.686 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:48.704 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:25:48.704 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:48.761 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:25:48.761 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:48.818 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:48.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.980+0000 7fba740ee700 1 Processor -- start 2024-01-15T11:25:48.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.980+0000 7fba740ee700 1 -- start start 2024-01-15T11:25:48.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.981+0000 7fba740ee700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fba6c15e040 0x7fba6c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:48.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.981+0000 7fba740ee700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fba6c0d02a0 0x7fba6c0d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:48.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.981+0000 7fba740ee700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c0700f0 con 0x7fba6c0d1ba0 2024-01-15T11:25:48.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.981+0000 7fba740ee700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba6c06f610 con 0x7fba6c15e040 2024-01-15T11:25:48.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.981+0000 7fba740ee700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c06f8b0 con 0x7fba6c0d02a0 2024-01-15T11:25:48.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.981+0000 7fba71e8a700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fba6c15e040 0x7fba6c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:48.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.981+0000 7fba71689700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fba6c0d1ba0 0x7fba6c0d1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:55152/0 (socket says 172.21.15.113:55152) 2024-01-15T11:25:48.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.981+0000 7fba7268b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fba6c0d02a0 0x7fba6c0d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:48.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.981+0000 7fba71689700 1 -- 172.21.15.113:0/616259683 learned_addr learned my addr 172.21.15.113:0/616259683 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba70e88700 1 -- 172.21.15.113:0/616259683 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706172075 0 0) 0x7fba5c001110 con 0x7fba6c0d1ba0 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba70e88700 1 -- 172.21.15.113:0/616259683 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba580035b0 con 0x7fba6c0d1ba0 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba71e8a700 1 -- 172.21.15.113:0/616259683 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fba6c0d02a0 msgr2=0x7fba6c0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba71e8a700 1 --2- 172.21.15.113:0/616259683 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fba6c0d02a0 0x7fba6c0d06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba71e8a700 1 -- 172.21.15.113:0/616259683 >> v1:172.21.15.141:6789/0 conn(0x7fba6c0d1ba0 legacy=0x7fba6c0d1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba71e8a700 1 -- 172.21.15.113:0/616259683 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c0cfdc0 con 0x7fba6c15e040 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba7268b700 1 --2- 172.21.15.113:0/616259683 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fba6c0d02a0 0x7fba6c0d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba71e8a700 1 --2- 172.21.15.113:0/616259683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba6c15e040 0x7fba6c06ec90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fba68009840 tx=0x7fba68009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff7e46d878c51c4a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba70e88700 1 -- 172.21.15.113:0/616259683 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68011830 con 0x7fba6c15e040 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba70e88700 1 -- 172.21.15.113:0/616259683 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba68011990 con 0x7fba6c15e040 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba70e88700 1 -- 172.21.15.113:0/616259683 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68011d20 con 0x7fba6c15e040 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba740ee700 1 -- 172.21.15.113:0/616259683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba6c15e040 msgr2=0x7fba6c06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.982+0000 7fba740ee700 1 --2- 172.21.15.113:0/616259683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba6c15e040 0x7fba6c06ec90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fba68009840 tx=0x7fba68009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.983+0000 7fba740ee700 1 -- 172.21.15.113:0/616259683 shutdown_connections 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.983+0000 7fba740ee700 1 --2- 172.21.15.113:0/616259683 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fba6c0d02a0 0x7fba6c0d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.983+0000 7fba740ee700 1 --2- 172.21.15.113:0/616259683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba6c15e040 0x7fba6c06ec90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.983+0000 7fba740ee700 1 -- 172.21.15.113:0/616259683 >> 172.21.15.113:0/616259683 conn(0x7fba6c073130 msgr2=0x7fba6c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:48.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.983+0000 7fba740ee700 1 -- 172.21.15.113:0/616259683 shutdown_connections 2024-01-15T11:25:48.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.983+0000 7fba740ee700 1 -- 172.21.15.113:0/616259683 wait complete. 2024-01-15T11:25:48.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.984+0000 7fba740ee700 1 Processor -- start 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.984+0000 7fba740ee700 1 -- start start 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.984+0000 7fba740ee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0d02a0 0x7fba6c0c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba740ee700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba6c0d1ba0 0x7fba6c160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba740ee700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fba6c15e040 0x7fba6c0b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba740ee700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c15e730 con 0x7fba6c0d02a0 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba740ee700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c161510 con 0x7fba6c0d1ba0 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba740ee700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c0c8430 con 0x7fba6c15e040 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba71e8a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0d02a0 0x7fba6c0c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba71e8a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0d02a0 0x7fba6c0c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44860/0 (socket says 172.21.15.113:44860) 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba71e8a700 1 -- 172.21.15.113:0/1659942203 learned_addr learned my addr 172.21.15.113:0/1659942203 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba71689700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba6c0d1ba0 0x7fba6c160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba7268b700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fba6c15e040 0x7fba6c0b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba71e8a700 1 -- 172.21.15.113:0/1659942203 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fba6c15e040 msgr2=0x7fba6c0b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba71e8a700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fba6c15e040 0x7fba6c0b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba71e8a700 1 -- 172.21.15.113:0/1659942203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba6c0d1ba0 msgr2=0x7fba6c160fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba71e8a700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba6c0d1ba0 0x7fba6c160fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba71e8a700 1 -- 172.21.15.113:0/1659942203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba64009a70 con 0x7fba6c0d02a0 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba71689700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba6c0d1ba0 0x7fba6c160fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba71e8a700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0d02a0 0x7fba6c0c5760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fba68009fd0 tx=0x7fba68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba627fc700 1 -- 172.21.15.113:0/1659942203 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68004550 con 0x7fba6c0d02a0 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba627fc700 1 -- 172.21.15.113:0/1659942203 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6801e070 con 0x7fba6c0d02a0 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba627fc700 1 -- 172.21.15.113:0/1659942203 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba680038e0 con 0x7fba6c0d02a0 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba740ee700 1 -- 172.21.15.113:0/1659942203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba680094a0 con 0x7fba6c0d02a0 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.985+0000 7fba740ee700 1 -- 172.21.15.113:0/1659942203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6c15ec60 con 0x7fba6c0d02a0 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.986+0000 7fba740ee700 1 -- 172.21.15.113:0/1659942203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6c067f60 con 0x7fba6c0d02a0 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.987+0000 7fba627fc700 1 -- 172.21.15.113:0/1659942203 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fba68003a40 con 0x7fba6c0d02a0 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.987+0000 7fba627fc700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fba5006b350 0x7fba5006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:48.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.987+0000 7fba627fc700 1 -- 172.21.15.113:0/1659942203 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba6808a8f0 con 0x7fba6c0d02a0 2024-01-15T11:25:48.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.987+0000 7fba71689700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fba5006b350 0x7fba5006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:48.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.988+0000 7fba71689700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fba5006b350 0x7fba5006d810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fba5c007fd0 tx=0x7fba5c007c40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:48.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:48.992+0000 7fba627fc700 1 -- 172.21.15.113:0/1659942203 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba680557f0 con 0x7fba6c0d02a0 2024-01-15T11:25:49.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.184+0000 7fba740ee700 1 -- 172.21.15.113:0/1659942203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba6c0d2050 con 0x7fba6c0d02a0 2024-01-15T11:25:49.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.186+0000 7fba627fc700 1 -- 172.21.15.113:0/1659942203 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7fba68015090 con 0x7fba6c0d02a0 2024-01-15T11:25:49.185 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:49.185 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:49.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.188+0000 7fba740ee700 1 -- 172.21.15.113:0/1659942203 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fba5006b350 msgr2=0x7fba5006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.188+0000 7fba740ee700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fba5006b350 0x7fba5006d810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fba5c007fd0 tx=0x7fba5c007c40 comp rx=0 tx=0).stop 2024-01-15T11:25:49.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.188+0000 7fba740ee700 1 -- 172.21.15.113:0/1659942203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0d02a0 msgr2=0x7fba6c0c5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.188+0000 7fba740ee700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0d02a0 0x7fba6c0c5760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fba68009fd0 tx=0x7fba68004750 comp rx=0 tx=0).stop 2024-01-15T11:25:49.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.188+0000 7fba740ee700 1 -- 172.21.15.113:0/1659942203 shutdown_connections 2024-01-15T11:25:49.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.188+0000 7fba740ee700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fba5006b350 0x7fba5006d810 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.188+0000 7fba740ee700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fba6c15e040 0x7fba6c0b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.188+0000 7fba740ee700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0d02a0 0x7fba6c0c5760 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.188+0000 7fba740ee700 1 --2- 172.21.15.113:0/1659942203 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fba6c0d1ba0 0x7fba6c160fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.188+0000 7fba740ee700 1 -- 172.21.15.113:0/1659942203 >> 172.21.15.113:0/1659942203 conn(0x7fba6c073130 msgr2=0x7fba6c0cd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:49.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.188+0000 7fba740ee700 1 -- 172.21.15.113:0/1659942203 shutdown_connections 2024-01-15T11:25:49.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.188+0000 7fba740ee700 1 -- 172.21.15.113:0/1659942203 wait complete. 2024-01-15T11:25:49.199 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T11:25:13.398220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T11:25:14.589439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:49.199 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:49.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.324+0000 7ff9b6f32700 1 Processor -- start 2024-01-15T11:25:49.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.324+0000 7ff9b6f32700 1 -- start start 2024-01-15T11:25:49.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9b6f32700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015ae00 0x7ff9b006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9b6f32700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b00b3d60 con 0x7ff9b00d05b0 2024-01-15T11:25:49.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9b6f32700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b00b4040 con 0x7ff9b00add40 2024-01-15T11:25:49.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9b6f32700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b015f8b0 con 0x7ff9b015ae00 2024-01-15T11:25:49.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9b54cf700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ff9b00d05b0 0x7ff9b00b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42250/0 (socket says 172.21.15.113:42250) 2024-01-15T11:25:49.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9b54cf700 1 -- 172.21.15.113:0/2058403182 learned_addr learned my addr 172.21.15.113:0/2058403182 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:49.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9b4cce700 1 --2- 172.21.15.113:0/2058403182 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015ae00 0x7ff9b006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247266625 0 0) 0x7ff9b00b3d60 con 0x7ff9b00d05b0 2024-01-15T11:25:49.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a00035b0 con 0x7ff9b00d05b0 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379479194 0 0) 0x7ff9b00b4040 con 0x7ff9b00add40 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b00b3d60 con 0x7ff9b00add40 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1404774850 0 0) 0x7ff9a00035b0 con 0x7ff9b00d05b0 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015ae00 msgr2=0x7ff9b006dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9af7fe700 1 --2- 172.21.15.113:0/2058403182 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015ae00 0x7ff9b006dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 >> v1:172.21.15.141:6789/0 conn(0x7ff9b00add40 legacy=0x7ff9b00cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b00b5be0 con 0x7ff9b00d05b0 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.325+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7ff9a80019f0 con 0x7ff9b00d05b0 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 >> v1:172.21.15.113:6789/0 conn(0x7ff9b00d05b0 legacy=0x7ff9b00b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9af7fe700 1 --2- 172.21.15.113:0/2058403182 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015ae00 0x7ff9a0003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9af7fe700 1 --2- 172.21.15.113:0/2058403182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9a0004170 0x7ff9a0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9af7fe700 1 --2- 172.21.15.113:0/2058403182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9a0004d90 0x7ff9a0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b015f8b0 con 0x7ff9a0004170 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a000bbd0 con 0x7ff9a0004d90 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a000bd90 con 0x7ff9b015ae00 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9affff700 1 --2- 172.21.15.113:0/2058403182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9a0004170 0x7ff9a0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9b4cce700 1 --2- 172.21.15.113:0/2058403182 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015ae00 0x7ff9a0003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9b54cf700 1 --2- 172.21.15.113:0/2058403182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9a0004d90 0x7ff9a0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9affff700 1 -- 172.21.15.113:0/2058403182 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015ae00 msgr2=0x7ff9a0003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9affff700 1 --2- 172.21.15.113:0/2058403182 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015ae00 0x7ff9a0003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9affff700 1 -- 172.21.15.113:0/2058403182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9a0004d90 msgr2=0x7ff9a0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9affff700 1 --2- 172.21.15.113:0/2058403182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9a0004d90 0x7ff9a0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9affff700 1 -- 172.21.15.113:0/2058403182 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a00035b0 con 0x7ff9a0004170 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9affff700 1 --2- 172.21.15.113:0/2058403182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9a0004170 0x7ff9a0004850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff99c001940 tx=0x7ff9a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de11ceb54e14c10 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c001430 con 0x7ff9a0004170 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c001d70 con 0x7ff9a0004170 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.326+0000 7ff9af7fe700 1 -- 172.21.15.113:0/2058403182 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c005510 con 0x7ff9a0004170 2024-01-15T11:25:49.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.327+0000 7ff9b6f32700 1 -- 172.21.15.113:0/2058403182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9a0004170 msgr2=0x7ff9a0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.327+0000 7ff9b6f32700 1 --2- 172.21.15.113:0/2058403182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9a0004170 0x7ff9a0004850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff99c001940 tx=0x7ff9a0005880 comp rx=0 tx=0).stop 2024-01-15T11:25:49.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.327+0000 7ff9b54cf700 1 -- 172.21.15.113:0/2058403182 reap_dead start 2024-01-15T11:25:49.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.327+0000 7ff9b6f32700 1 -- 172.21.15.113:0/2058403182 shutdown_connections 2024-01-15T11:25:49.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.327+0000 7ff9b6f32700 1 -- 172.21.15.113:0/2058403182 >> 172.21.15.113:0/2058403182 conn(0x7ff9b0072b20 msgr2=0x7ff9b006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:49.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.327+0000 7ff9b6f32700 1 -- 172.21.15.113:0/2058403182 shutdown_connections 2024-01-15T11:25:49.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.327+0000 7ff9b6f32700 1 -- 172.21.15.113:0/2058403182 wait complete. 2024-01-15T11:25:49.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.328+0000 7ff9b6f32700 1 Processor -- start 2024-01-15T11:25:49.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.328+0000 7ff9b6f32700 1 -- start start 2024-01-15T11:25:49.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b6f32700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9a0004170 0x7ff9b0155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b6f32700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9b015dc80 0x7ff9b01f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b6f32700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015cc50 0x7ff9b01523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b6f32700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b01f1090 con 0x7ff9a0004170 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b6f32700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b01f1370 con 0x7ff9b015dc80 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b6f32700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b015c450 con 0x7ff9b015cc50 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b4cce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9a0004170 0x7ff9b0155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b4cce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9a0004170 0x7ff9b0155730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44876/0 (socket says 172.21.15.113:44876) 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b4cce700 1 -- 172.21.15.113:0/635262699 learned_addr learned my addr 172.21.15.113:0/635262699 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b54cf700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015cc50 0x7ff9b01523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9affff700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9b015dc80 0x7ff9b01f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b4cce700 1 -- 172.21.15.113:0/635262699 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015cc50 msgr2=0x7ff9b01523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b4cce700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015cc50 0x7ff9b01523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b4cce700 1 -- 172.21.15.113:0/635262699 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9b015dc80 msgr2=0x7ff9b01f06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b4cce700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9b015dc80 0x7ff9b01f06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9b4cce700 1 -- 172.21.15.113:0/635262699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b015c730 con 0x7ff9a0004170 2024-01-15T11:25:49.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.329+0000 7ff9affff700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9b015dc80 0x7ff9b01f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:49.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.330+0000 7ff9b4cce700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9a0004170 0x7ff9b0155730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4011aa0 tx=0x7ff9a4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:49.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.330+0000 7ff9ad7fa700 1 -- 172.21.15.113:0/635262699 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a400f980 con 0x7ff9a0004170 2024-01-15T11:25:49.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.330+0000 7ff9ad7fa700 1 -- 172.21.15.113:0/635262699 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a400fae0 con 0x7ff9a0004170 2024-01-15T11:25:49.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.330+0000 7ff9b6f32700 1 -- 172.21.15.113:0/635262699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b01f1650 con 0x7ff9a0004170 2024-01-15T11:25:49.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.330+0000 7ff9ad7fa700 1 -- 172.21.15.113:0/635262699 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a400fe90 con 0x7ff9a0004170 2024-01-15T11:25:49.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.330+0000 7ff9b6f32700 1 -- 172.21.15.113:0/635262699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b0155e50 con 0x7ff9a0004170 2024-01-15T11:25:49.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.331+0000 7ff9b6f32700 1 -- 172.21.15.113:0/635262699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b0069020 con 0x7ff9a0004170 2024-01-15T11:25:49.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.331+0000 7ff9ad7fa700 1 -- 172.21.15.113:0/635262699 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff9a4014440 con 0x7ff9a0004170 2024-01-15T11:25:49.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.331+0000 7ff9ad7fa700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff99406b350 0x7ff99406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.332+0000 7ff9ad7fa700 1 -- 172.21.15.113:0/635262699 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9a408a060 con 0x7ff9a0004170 2024-01-15T11:25:49.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.332+0000 7ff9affff700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff99406b350 0x7ff99406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.332+0000 7ff9affff700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff99406b350 0x7ff99406d810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff99c015040 tx=0x7ff99c00a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:49.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.336+0000 7ff9ad7fa700 1 -- 172.21.15.113:0/635262699 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9a4055020 con 0x7ff9a0004170 2024-01-15T11:25:49.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.524+0000 7ff9b6f32700 1 -- 172.21.15.113:0/635262699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff9b015c8c0 con 0x7ff9a0004170 2024-01-15T11:25:49.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.524+0000 7ff9ad7fa700 1 -- 172.21.15.113:0/635262699 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff9a4058640 con 0x7ff9a0004170 2024-01-15T11:25:49.523 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:49.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.526+0000 7ff9b6f32700 1 -- 172.21.15.113:0/635262699 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff99406b350 msgr2=0x7ff99406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.526+0000 7ff9b6f32700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff99406b350 0x7ff99406d810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff99c015040 tx=0x7ff99c00a000 comp rx=0 tx=0).stop 2024-01-15T11:25:49.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.527+0000 7ff9b6f32700 1 -- 172.21.15.113:0/635262699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9a0004170 msgr2=0x7ff9b0155730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.527+0000 7ff9b6f32700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9a0004170 0x7ff9b0155730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4011aa0 tx=0x7ff9a4011e60 comp rx=0 tx=0).stop 2024-01-15T11:25:49.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.527+0000 7ff9b6f32700 1 -- 172.21.15.113:0/635262699 shutdown_connections 2024-01-15T11:25:49.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.527+0000 7ff9b6f32700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff99406b350 0x7ff99406d810 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.527+0000 7ff9b6f32700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff9b015cc50 0x7ff9b01523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.527+0000 7ff9b6f32700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9a0004170 0x7ff9b0155730 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.527+0000 7ff9b6f32700 1 --2- 172.21.15.113:0/635262699 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff9b015dc80 0x7ff9b01f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.527+0000 7ff9b6f32700 1 -- 172.21.15.113:0/635262699 >> 172.21.15.113:0/635262699 conn(0x7ff9b0072b20 msgr2=0x7ff9b00b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:49.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.527+0000 7ff9b6f32700 1 -- 172.21.15.113:0/635262699 shutdown_connections 2024-01-15T11:25:49.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.527+0000 7ff9b6f32700 1 -- 172.21.15.113:0/635262699 wait complete. 2024-01-15T11:25:49.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T11:25:49.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.663+0000 7fbaa4e17700 1 Processor -- start 2024-01-15T11:25:49.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.663+0000 7fbaa4e17700 1 -- start start 2024-01-15T11:25:49.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.664+0000 7fbaa4e17700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbaa00d05b0 0x7fbaa006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.664+0000 7fbaa4e17700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fbaa00add40 0x7fbaa015f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.664+0000 7fbaa4e17700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa00b3d60 con 0x7fbaa015ae00 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.664+0000 7fbaa4e17700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa015f8b0 con 0x7fbaa00add40 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.664+0000 7fbaa4e17700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa015b1e0 con 0x7fbaa00d05b0 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.664+0000 7fba9e59c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fbaa015ae00 0x7fbaa00cfea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42254/0 (socket says 172.21.15.113:42254) 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.664+0000 7fba9e59c700 1 -- 172.21.15.113:0/4078819586 learned_addr learned my addr 172.21.15.113:0/4078819586 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.664+0000 7fba9ed9d700 1 --2- 172.21.15.113:0/4078819586 >> v2:172.21.15.141:3300/0 conn(0x7fbaa00add40 0x7fbaa015f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.664+0000 7fba9dd9b700 1 --2- 172.21.15.113:0/4078819586 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbaa00d05b0 0x7fbaa006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.664+0000 7fba9d59a700 1 -- 172.21.15.113:0/4078819586 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172910114 0 0) 0x7fbaa00b3d60 con 0x7fbaa015ae00 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.664+0000 7fba9d59a700 1 -- 172.21.15.113:0/4078819586 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba880035b0 con 0x7fbaa015ae00 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.665+0000 7fba9ed9d700 1 -- 172.21.15.113:0/4078819586 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbaa00d05b0 msgr2=0x7fbaa006dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.665+0000 7fba9ed9d700 1 --2- 172.21.15.113:0/4078819586 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbaa00d05b0 0x7fbaa006dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.665+0000 7fba9ed9d700 1 -- 172.21.15.113:0/4078819586 >> v1:172.21.15.113:6789/0 conn(0x7fbaa015ae00 legacy=0x7fbaa00cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.665+0000 7fba9ed9d700 1 -- 172.21.15.113:0/4078819586 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00b4040 con 0x7fbaa00add40 2024-01-15T11:25:49.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.665+0000 7fba9ed9d700 1 --2- 172.21.15.113:0/4078819586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbaa00add40 0x7fbaa015f280 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fba98009df0 tx=0x7fba9800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7887359f8787e27b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:49.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.665+0000 7fba9d59a700 1 -- 172.21.15.113:0/4078819586 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9800e070 con 0x7fbaa00add40 2024-01-15T11:25:49.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.665+0000 7fba9d59a700 1 -- 172.21.15.113:0/4078819586 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9800baa0 con 0x7fbaa00add40 2024-01-15T11:25:49.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.665+0000 7fba9d59a700 1 -- 172.21.15.113:0/4078819586 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9800be50 con 0x7fbaa00add40 2024-01-15T11:25:49.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.665+0000 7fbaa4e17700 1 -- 172.21.15.113:0/4078819586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbaa00add40 msgr2=0x7fbaa015f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.666+0000 7fbaa4e17700 1 --2- 172.21.15.113:0/4078819586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbaa00add40 0x7fbaa015f280 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fba98009df0 tx=0x7fba9800b330 comp rx=0 tx=0).stop 2024-01-15T11:25:49.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.666+0000 7fbaa4e17700 1 -- 172.21.15.113:0/4078819586 shutdown_connections 2024-01-15T11:25:49.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.666+0000 7fbaa4e17700 1 --2- 172.21.15.113:0/4078819586 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbaa00add40 0x7fbaa015f280 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.666+0000 7fbaa4e17700 1 --2- 172.21.15.113:0/4078819586 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbaa00d05b0 0x7fbaa006dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.666+0000 7fbaa4e17700 1 -- 172.21.15.113:0/4078819586 >> 172.21.15.113:0/4078819586 conn(0x7fbaa0072b20 msgr2=0x7fbaa006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:49.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.666+0000 7fbaa4e17700 1 -- 172.21.15.113:0/4078819586 shutdown_connections 2024-01-15T11:25:49.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.666+0000 7fbaa4e17700 1 -- 172.21.15.113:0/4078819586 wait complete. 2024-01-15T11:25:49.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.667+0000 7fbaa4e17700 1 Processor -- start 2024-01-15T11:25:49.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.667+0000 7fbaa4e17700 1 -- start start 2024-01-15T11:25:49.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fbaa4e17700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbaa00add40 0x7fbaa0074200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fbaa4e17700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa00d05b0 0x7fbaa00cf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fbaa4e17700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbaa015ae00 0x7fbaa014b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fbaa4e17700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa014e1f0 con 0x7fbaa00d05b0 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fbaa4e17700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa014d5b0 con 0x7fbaa00add40 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fbaa4e17700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa014d850 con 0x7fbaa015ae00 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba9dd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa00d05b0 0x7fbaa00cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba9dd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa00d05b0 0x7fbaa00cf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44890/0 (socket says 172.21.15.113:44890) 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba9dd9b700 1 -- 172.21.15.113:0/2455182611 learned_addr learned my addr 172.21.15.113:0/2455182611 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba9e59c700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbaa00add40 0x7fbaa0074200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba9dd9b700 1 -- 172.21.15.113:0/2455182611 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbaa015ae00 msgr2=0x7fbaa014b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba9dd9b700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbaa015ae00 0x7fbaa014b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba9dd9b700 1 -- 172.21.15.113:0/2455182611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbaa00add40 msgr2=0x7fbaa0074200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba9dd9b700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbaa00add40 0x7fbaa0074200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba9dd9b700 1 -- 172.21.15.113:0/2455182611 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba98009a70 con 0x7fbaa00d05b0 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba9e59c700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbaa00add40 0x7fbaa0074200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba9dd9b700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa00d05b0 0x7fbaa00cf340 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fba9000e9b0 tx=0x7fba9000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba8effd700 1 -- 172.21.15.113:0/2455182611 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90009710 con 0x7fbaa00d05b0 2024-01-15T11:25:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba8effd700 1 -- 172.21.15.113:0/2455182611 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9000c850 con 0x7fbaa00d05b0 2024-01-15T11:25:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fbaa4e17700 1 -- 172.21.15.113:0/2455182611 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa014db30 con 0x7fbaa00d05b0 2024-01-15T11:25:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.668+0000 7fba8effd700 1 -- 172.21.15.113:0/2455182611 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9000cbc0 con 0x7fbaa00d05b0 2024-01-15T11:25:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.669+0000 7fbaa4e17700 1 -- 172.21.15.113:0/2455182611 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa0074740 con 0x7fbaa00d05b0 2024-01-15T11:25:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.669+0000 7fbaa4e17700 1 -- 172.21.15.113:0/2455182611 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa014e330 con 0x7fbaa00d05b0 2024-01-15T11:25:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.670+0000 7fba8effd700 1 -- 172.21.15.113:0/2455182611 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fba9000cd20 con 0x7fbaa00d05b0 2024-01-15T11:25:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.670+0000 7fba8effd700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fba8006b350 0x7fba8006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.670+0000 7fba8effd700 1 -- 172.21.15.113:0/2455182611 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba90089ce0 con 0x7fbaa00d05b0 2024-01-15T11:25:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.670+0000 7fba9e59c700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fba8006b350 0x7fba8006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.671+0000 7fba9e59c700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fba8006b350 0x7fba8006d810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fba940011e0 tx=0x7fba94009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:49.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.675+0000 7fba8effd700 1 -- 172.21.15.113:0/2455182611 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbaa014e330 con 0x7fbaa00d05b0 2024-01-15T11:25:49.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.867+0000 7fbaa4e17700 1 -- 172.21.15.113:0/2455182611 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbaa014e330 con 0x7fbaa00d05b0 2024-01-15T11:25:49.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.867+0000 7fba8effd700 1 -- 172.21.15.113:0/2455182611 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbaa014e330 con 0x7fbaa00d05b0 2024-01-15T11:25:49.866 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-15T11:25:49.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.869+0000 7fbaa4e17700 1 -- 172.21.15.113:0/2455182611 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fba8006b350 msgr2=0x7fba8006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.869+0000 7fbaa4e17700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fba8006b350 0x7fba8006d810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fba940011e0 tx=0x7fba94009040 comp rx=0 tx=0).stop 2024-01-15T11:25:49.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.869+0000 7fbaa4e17700 1 -- 172.21.15.113:0/2455182611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa00d05b0 msgr2=0x7fbaa00cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:49.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.869+0000 7fbaa4e17700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa00d05b0 0x7fbaa00cf340 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fba9000e9b0 tx=0x7fba9000ed70 comp rx=0 tx=0).stop 2024-01-15T11:25:49.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.869+0000 7fbaa4e17700 1 -- 172.21.15.113:0/2455182611 shutdown_connections 2024-01-15T11:25:49.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.869+0000 7fbaa4e17700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fba8006b350 0x7fba8006d810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.869+0000 7fbaa4e17700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fbaa015ae00 0x7fbaa014b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.869+0000 7fbaa4e17700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbaa00add40 0x7fbaa0074200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.869+0000 7fbaa4e17700 1 --2- 172.21.15.113:0/2455182611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa00d05b0 0x7fbaa00cf340 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:49.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.869+0000 7fbaa4e17700 1 -- 172.21.15.113:0/2455182611 >> 172.21.15.113:0/2455182611 conn(0x7fbaa0072b20 msgr2=0x7fbaa006f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:49.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.869+0000 7fbaa4e17700 1 -- 172.21.15.113:0/2455182611 shutdown_connections 2024-01-15T11:25:49.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:49.869+0000 7fbaa4e17700 1 -- 172.21.15.113:0/2455182611 wait complete. 2024-01-15T11:25:49.880 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T11:25:50.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.005+0000 7f35f4fcf700 1 Processor -- start 2024-01-15T11:25:50.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.006+0000 7f35f4fcf700 1 -- start start 2024-01-15T11:25:50.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.006+0000 7f35f4fcf700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f35f00cffc0 0x7f35f00b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.006+0000 7f35f4fcf700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f35f015f8d0 0x7f35f00ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.006+0000 7f35f4fcf700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00b3860 con 0x7f35f00d03a0 2024-01-15T11:25:50.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.006+0000 7f35f4fcf700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f00b2520 con 0x7f35f015f8d0 2024-01-15T11:25:50.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.006+0000 7f35f4fcf700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00b2800 con 0x7f35f00cffc0 2024-01-15T11:25:50.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.007+0000 7f35eed9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f35f015f8d0 0x7f35f00ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.007+0000 7f35eed9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f35f015f8d0 0x7f35f00ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44906/0 (socket says 172.21.15.113:44906) 2024-01-15T11:25:50.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.007+0000 7f35eed9d700 1 -- 172.21.15.113:0/2145677873 learned_addr learned my addr 172.21.15.113:0/2145677873 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:50.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.007+0000 7f35ee59c700 1 --2- 172.21.15.113:0/2145677873 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f35f00cffc0 0x7f35f00b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.007+0000 7f35eed9d700 1 -- 172.21.15.113:0/2145677873 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f35f00cffc0 msgr2=0x7f35f00b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:50.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.007+0000 7f35eed9d700 1 --2- 172.21.15.113:0/2145677873 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f35f00cffc0 0x7f35f00b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.007+0000 7f35eed9d700 1 -- 172.21.15.113:0/2145677873 >> v1:172.21.15.141:6789/0 conn(0x7f35f00d03a0 legacy=0x7f35f015f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.007+0000 7f35eed9d700 1 -- 172.21.15.113:0/2145677873 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00b2ae0 con 0x7f35f015f8d0 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.007+0000 7f35eed9d700 1 --2- 172.21.15.113:0/2145677873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35f015f8d0 0x7f35f00ccd10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009e50 tx=0x7f35e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b450e511849e3a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.007+0000 7f35ed59a700 1 -- 172.21.15.113:0/2145677873 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e800e070 con 0x7f35f015f8d0 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.007+0000 7f35ed59a700 1 -- 172.21.15.113:0/2145677873 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e800b9c0 con 0x7f35f015f8d0 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.007+0000 7f35ed59a700 1 -- 172.21.15.113:0/2145677873 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e800bd70 con 0x7f35f015f8d0 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.008+0000 7f35f4fcf700 1 -- 172.21.15.113:0/2145677873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35f015f8d0 msgr2=0x7f35f00ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.008+0000 7f35f4fcf700 1 --2- 172.21.15.113:0/2145677873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35f015f8d0 0x7f35f00ccd10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009e50 tx=0x7f35e800b330 comp rx=0 tx=0).stop 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.008+0000 7f35f4fcf700 1 -- 172.21.15.113:0/2145677873 shutdown_connections 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.008+0000 7f35f4fcf700 1 --2- 172.21.15.113:0/2145677873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35f015f8d0 0x7f35f00ccd10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.008+0000 7f35f4fcf700 1 --2- 172.21.15.113:0/2145677873 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f35f00cffc0 0x7f35f00b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.008+0000 7f35f4fcf700 1 -- 172.21.15.113:0/2145677873 >> 172.21.15.113:0/2145677873 conn(0x7f35f00d1e50 msgr2=0x7f35f00d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.008+0000 7f35f4fcf700 1 -- 172.21.15.113:0/2145677873 shutdown_connections 2024-01-15T11:25:50.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.008+0000 7f35f4fcf700 1 -- 172.21.15.113:0/2145677873 wait complete. 2024-01-15T11:25:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.009+0000 7f35f4fcf700 1 Processor -- start 2024-01-15T11:25:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.010+0000 7f35f4fcf700 1 -- start start 2024-01-15T11:25:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.010+0000 7f35f4fcf700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35f00cffc0 0x7f35f00b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.010+0000 7f35f4fcf700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f35f00d03a0 0x7f35f01f1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.010+0000 7f35f4fcf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35f015f8d0 0x7f35f00c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.010+0000 7f35f4fcf700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f015ba80 con 0x7f35f015f8d0 2024-01-15T11:25:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35f4fcf700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00c5ec0 con 0x7f35f00cffc0 2024-01-15T11:25:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35f4fcf700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00c60f0 con 0x7f35f00d03a0 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35eed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35f015f8d0 0x7f35f00c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35eed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35f015f8d0 0x7f35f00c5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44910/0 (socket says 172.21.15.113:44910) 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35eed9d700 1 -- 172.21.15.113:0/570546318 learned_addr learned my addr 172.21.15.113:0/570546318 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35ee59c700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35f00cffc0 0x7f35f00b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35edd9b700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f35f00d03a0 0x7f35f01f1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35eed9d700 1 -- 172.21.15.113:0/570546318 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f35f00d03a0 msgr2=0x7f35f01f1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35eed9d700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f35f00d03a0 0x7f35f01f1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35eed9d700 1 -- 172.21.15.113:0/570546318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35f00cffc0 msgr2=0x7f35f00b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35eed9d700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35f00cffc0 0x7f35f00b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35eed9d700 1 -- 172.21.15.113:0/570546318 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e8009b30 con 0x7f35f015f8d0 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35edd9b700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f35f00d03a0 0x7f35f01f1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35eed9d700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35f015f8d0 0x7f35f00c5980 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f35f0072870 tx=0x7f35e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:50.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35e6ffd700 1 -- 172.21.15.113:0/570546318 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e80048b0 con 0x7f35f015f8d0 2024-01-15T11:25:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35e6ffd700 1 -- 172.21.15.113:0/570546318 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e80099b0 con 0x7f35f015f8d0 2024-01-15T11:25:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35f4fcf700 1 -- 172.21.15.113:0/570546318 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f00c63d0 con 0x7f35f015f8d0 2024-01-15T11:25:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35e6ffd700 1 -- 172.21.15.113:0/570546318 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e801b670 con 0x7f35f015f8d0 2024-01-15T11:25:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.011+0000 7f35f4fcf700 1 -- 172.21.15.113:0/570546318 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f00c6990 con 0x7f35f015f8d0 2024-01-15T11:25:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.012+0000 7f35f4fcf700 1 -- 172.21.15.113:0/570546318 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f0073670 con 0x7f35f015f8d0 2024-01-15T11:25:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.013+0000 7f35e6ffd700 1 -- 172.21.15.113:0/570546318 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f35e801b7d0 con 0x7f35f015f8d0 2024-01-15T11:25:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.013+0000 7f35e6ffd700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f35d006b350 0x7f35d006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.013+0000 7f35e6ffd700 1 -- 172.21.15.113:0/570546318 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35e808b470 con 0x7f35f015f8d0 2024-01-15T11:25:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.013+0000 7f35ee59c700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f35d006b350 0x7f35d006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.013+0000 7f35ee59c700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f35d006b350 0x7f35d006d810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f35e0006b20 tx=0x7f35e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:50.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.017+0000 7f35e6ffd700 1 -- 172.21.15.113:0/570546318 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35e80562f0 con 0x7f35f015f8d0 2024-01-15T11:25:50.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.210+0000 7f35f4fcf700 1 -- 172.21.15.113:0/570546318 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f35f0069020 con 0x7f35f015f8d0 2024-01-15T11:25:50.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.211+0000 7f35e6ffd700 1 -- 172.21.15.113:0/570546318 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f35e8059910 con 0x7f35f015f8d0 2024-01-15T11:25:50.210 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-15T11:25:50.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.213+0000 7f35f4fcf700 1 -- 172.21.15.113:0/570546318 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f35d006b350 msgr2=0x7f35d006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.213+0000 7f35f4fcf700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f35d006b350 0x7f35d006d810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f35e0006b20 tx=0x7f35e0008040 comp rx=0 tx=0).stop 2024-01-15T11:25:50.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.213+0000 7f35f4fcf700 1 -- 172.21.15.113:0/570546318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35f015f8d0 msgr2=0x7f35f00c5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.213+0000 7f35f4fcf700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35f015f8d0 0x7f35f00c5980 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f35f0072870 tx=0x7f35e8004ad0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.213+0000 7f35f4fcf700 1 -- 172.21.15.113:0/570546318 shutdown_connections 2024-01-15T11:25:50.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.213+0000 7f35f4fcf700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f35d006b350 0x7f35d006d810 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.213+0000 7f35f4fcf700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35f015f8d0 0x7f35f00c5980 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.213+0000 7f35f4fcf700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f35f00cffc0 0x7f35f00b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.213+0000 7f35f4fcf700 1 --2- 172.21.15.113:0/570546318 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f35f00d03a0 0x7f35f01f1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.213+0000 7f35f4fcf700 1 -- 172.21.15.113:0/570546318 >> 172.21.15.113:0/570546318 conn(0x7f35f00d1e50 msgr2=0x7f35f015d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:50.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.213+0000 7f35f4fcf700 1 -- 172.21.15.113:0/570546318 shutdown_connections 2024-01-15T11:25:50.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.213+0000 7f35f4fcf700 1 -- 172.21.15.113:0/570546318 wait complete. 2024-01-15T11:25:50.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T11:25:50.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.346+0000 7f6e83be6700 1 Processor -- start 2024-01-15T11:25:50.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.347+0000 7f6e83be6700 1 -- start start 2024-01-15T11:25:50.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.347+0000 7f6e83be6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e7c15bed0 0x7f6e7c0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.347+0000 7f6e83be6700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6e7c0d2160 0x7f6e7c0d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.347+0000 7f6e83be6700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7c0d0770 con 0x7f6e7c1603c0 2024-01-15T11:25:50.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.347+0000 7f6e83be6700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7c15c2b0 con 0x7f6e7c0d2160 2024-01-15T11:25:50.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.347+0000 7f6e83be6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0cd100 con 0x7f6e7c15bed0 2024-01-15T11:25:50.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e81181700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f6e7c1603c0 0x7f6e7c0d1a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42270/0 (socket says 172.21.15.113:42270) 2024-01-15T11:25:50.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e81181700 1 -- 172.21.15.113:0/3872375646 learned_addr learned my addr 172.21.15.113:0/3872375646 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:50.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161780472 0 0) 0x7f6e7c0d0770 con 0x7f6e7c1603c0 2024-01-15T11:25:50.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e81982700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e7c15bed0 0x7f6e7c0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e680035b0 con 0x7f6e7c1603c0 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e82183700 1 --2- 172.21.15.113:0/3872375646 >> v2:172.21.15.141:3300/0 conn(0x7f6e7c0d2160 0x7f6e7c0d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3472767518 0 0) 0x7f6e680035b0 con 0x7f6e7c1603c0 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e7c15bed0 msgr2=0x7f6e7c0b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e7c15bed0 0x7f6e7c0b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 >> v2:172.21.15.141:3300/0 conn(0x7f6e7c0d2160 msgr2=0x7f6e7c0d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 --2- 172.21.15.113:0/3872375646 >> v2:172.21.15.141:3300/0 conn(0x7f6e7c0d2160 0x7f6e7c0d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c0d09a0 con 0x7f6e7c1603c0 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f6e78001670 con 0x7f6e7c1603c0 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 >> v1:172.21.15.113:6789/0 conn(0x7f6e7c1603c0 legacy=0x7f6e7c0d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e68003a00 0x7f6e68003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e7c15bed0 0x7f6e68004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6e68004d90 0x7f6e68005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c15c2b0 con 0x7f6e68003a00 2024-01-15T11:25:50.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0cd100 con 0x7f6e68004d90 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6800ba10 con 0x7f6e7c15bed0 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e81982700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e68003a00 0x7f6e68003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e81181700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e7c15bed0 0x7f6e68004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.348+0000 7f6e82183700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6e68004d90 0x7f6e68005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e81982700 1 -- 172.21.15.113:0/3872375646 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e7c15bed0 msgr2=0x7f6e68004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e81982700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e7c15bed0 0x7f6e68004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e81982700 1 -- 172.21.15.113:0/3872375646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6e68004d90 msgr2=0x7f6e68005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e81982700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6e68004d90 0x7f6e68005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e81982700 1 -- 172.21.15.113:0/3872375646 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e680035b0 con 0x7f6e68003a00 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e82183700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6e68004d90 0x7f6e68005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e81982700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e68003a00 0x7f6e68003e20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009b60 tx=0x7f6e68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966bd82c2acda834 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c00e6c0 con 0x7f6e68003a00 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c00e820 con 0x7f6e68003a00 2024-01-15T11:25:50.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e80980700 1 -- 172.21.15.113:0/3872375646 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c00eb90 con 0x7f6e68003a00 2024-01-15T11:25:50.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e83be6700 1 -- 172.21.15.113:0/3872375646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e68003a00 msgr2=0x7f6e68003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e83be6700 1 --2- 172.21.15.113:0/3872375646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e68003a00 0x7f6e68003e20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009b60 tx=0x7f6e68005880 comp rx=0 tx=0).stop 2024-01-15T11:25:50.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e82183700 1 -- 172.21.15.113:0/3872375646 reap_dead start 2024-01-15T11:25:50.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e83be6700 1 -- 172.21.15.113:0/3872375646 shutdown_connections 2024-01-15T11:25:50.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e83be6700 1 -- 172.21.15.113:0/3872375646 >> 172.21.15.113:0/3872375646 conn(0x7f6e7c073130 msgr2=0x7f6e7c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:50.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e83be6700 1 -- 172.21.15.113:0/3872375646 shutdown_connections 2024-01-15T11:25:50.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.349+0000 7f6e83be6700 1 -- 172.21.15.113:0/3872375646 wait complete. 2024-01-15T11:25:50.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.351+0000 7f6e83be6700 1 Processor -- start 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.351+0000 7f6e83be6700 1 -- start start 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.351+0000 7f6e83be6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e68003a00 0x7f6e7c1500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.351+0000 7f6e83be6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6e7c15cdd0 0x7f6e7c0b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.351+0000 7f6e83be6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c15d250 0x7f6e7c0b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.351+0000 7f6e83be6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c150610 con 0x7f6e7c15d250 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.351+0000 7f6e83be6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c15d770 con 0x7f6e7c15cdd0 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.351+0000 7f6e83be6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c15da50 con 0x7f6e68003a00 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.351+0000 7f6e82183700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c15d250 0x7f6e7c0b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.351+0000 7f6e82183700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c15d250 0x7f6e7c0b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44928/0 (socket says 172.21.15.113:44928) 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.351+0000 7f6e82183700 1 -- 172.21.15.113:0/1312514852 learned_addr learned my addr 172.21.15.113:0/1312514852 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e81181700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6e7c15cdd0 0x7f6e7c0b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e81982700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e68003a00 0x7f6e7c1500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e82183700 1 -- 172.21.15.113:0/1312514852 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e68003a00 msgr2=0x7f6e7c1500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e82183700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e68003a00 0x7f6e7c1500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e82183700 1 -- 172.21.15.113:0/1312514852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6e7c15cdd0 msgr2=0x7f6e7c0b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e82183700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6e7c15cdd0 0x7f6e7c0b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e82183700 1 -- 172.21.15.113:0/1312514852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c15dd30 con 0x7f6e7c15d250 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e81181700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6e7c15cdd0 0x7f6e7c0b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e81982700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e68003a00 0x7f6e7c1500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e82183700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c15d250 0x7f6e7c0b3490 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c15f860 tx=0x7f6e740114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e727fc700 1 -- 172.21.15.113:0/1312514852 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7400e030 con 0x7f6e7c15d250 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e727fc700 1 -- 172.21.15.113:0/1312514852 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e74007e50 con 0x7f6e7c15d250 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e83be6700 1 -- 172.21.15.113:0/1312514852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7c15e010 con 0x7f6e7c15d250 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e727fc700 1 -- 172.21.15.113:0/1312514852 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74011e60 con 0x7f6e7c15d250 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.352+0000 7f6e83be6700 1 -- 172.21.15.113:0/1312514852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c06c9c0 con 0x7f6e7c15d250 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.353+0000 7f6e83be6700 1 -- 172.21.15.113:0/1312514852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7c15e530 con 0x7f6e7c15d250 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.353+0000 7f6e727fc700 1 -- 172.21.15.113:0/1312514852 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6e74016470 con 0x7f6e7c15d250 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.354+0000 7f6e727fc700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6e6006b350 0x7f6e6006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.354+0000 7f6e727fc700 1 -- 172.21.15.113:0/1312514852 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e7400f070 con 0x7f6e7c15d250 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.354+0000 7f6e81982700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6e6006b350 0x7f6e6006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.354+0000 7f6e81982700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6e6006b350 0x7f6e6006d810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0090b0 tx=0x7f6e6c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:50.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.359+0000 7f6e727fc700 1 -- 172.21.15.113:0/1312514852 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e7c15e530 con 0x7f6e7c15d250 2024-01-15T11:25:50.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.569+0000 7f6e83be6700 1 -- 172.21.15.113:0/1312514852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6e7c15e530 con 0x7f6e7c15d250 2024-01-15T11:25:50.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.569+0000 7f6e727fc700 1 -- 172.21.15.113:0/1312514852 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6e7c15e530 con 0x7f6e7c15d250 2024-01-15T11:25:50.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.571+0000 7f6e83be6700 1 -- 172.21.15.113:0/1312514852 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6e6006b350 msgr2=0x7f6e6006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.571+0000 7f6e83be6700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6e6006b350 0x7f6e6006d810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0090b0 tx=0x7f6e6c008030 comp rx=0 tx=0).stop 2024-01-15T11:25:50.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.571+0000 7f6e83be6700 1 -- 172.21.15.113:0/1312514852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c15d250 msgr2=0x7f6e7c0b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.571+0000 7f6e83be6700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c15d250 0x7f6e7c0b3490 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c15f860 tx=0x7f6e740114b0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.571+0000 7f6e83be6700 1 -- 172.21.15.113:0/1312514852 shutdown_connections 2024-01-15T11:25:50.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.571+0000 7f6e83be6700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6e6006b350 0x7f6e6006d810 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.571+0000 7f6e83be6700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c15d250 0x7f6e7c0b3490 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.571+0000 7f6e83be6700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6e68003a00 0x7f6e7c1500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.571+0000 7f6e83be6700 1 --2- 172.21.15.113:0/1312514852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6e7c15cdd0 0x7f6e7c0b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.571+0000 7f6e83be6700 1 -- 172.21.15.113:0/1312514852 >> 172.21.15.113:0/1312514852 conn(0x7f6e7c073130 msgr2=0x7f6e7c15f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:50.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.571+0000 7f6e83be6700 1 -- 172.21.15.113:0/1312514852 shutdown_connections 2024-01-15T11:25:50.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.571+0000 7f6e83be6700 1 -- 172.21.15.113:0/1312514852 wait complete. 2024-01-15T11:25:50.571 INFO:teuthology.orchestra.run.smithi113.stderr:listed 0 entries 2024-01-15T11:25:50.582 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-15T11:25:50.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:50.582 DEBUG:teuthology.orchestra.run.smithi113:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-01-15T11:25:50.628 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi113.front.sepia.ceph.com 2024-01-15T11:25:50.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:50.628 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T11:25:50.667 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:25:50.667 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:50.688 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:25:50.688 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:50.749 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:25:50.749 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:50.810 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:50.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.932+0000 7f1aadc68700 1 Processor -- start 2024-01-15T11:25:50.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.933+0000 7f1aadc68700 1 -- start start 2024-01-15T11:25:50.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.933+0000 7f1aadc68700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f1aa806eb70 0x7f1aa8070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.933+0000 7f1aadc68700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1aa815b140 0x7f1aa8071100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.933+0000 7f1aadc68700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa80b3000 con 0x7f1aa8158770 2024-01-15T11:25:50.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.933+0000 7f1aadc68700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa80b32e0 con 0x7f1aa815b140 2024-01-15T11:25:50.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.933+0000 7f1aadc68700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa80b35c0 con 0x7f1aa806eb70 2024-01-15T11:25:50.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.933+0000 7f1aa7fff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1aa815b140 0x7f1aa8071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.933+0000 7f1aa7fff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1aa815b140 0x7f1aa8071100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44938/0 (socket says 172.21.15.113:44938) 2024-01-15T11:25:50.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.933+0000 7f1aa7fff700 1 -- 172.21.15.113:0/4057905231 learned_addr learned my addr 172.21.15.113:0/4057905231 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:50.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.934+0000 7f1aa6ffd700 1 --2- 172.21.15.113:0/4057905231 >> v2:172.21.15.141:3300/0 conn(0x7f1aa806eb70 0x7f1aa8070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.934+0000 7f1aa7fff700 1 -- 172.21.15.113:0/4057905231 >> v2:172.21.15.141:3300/0 conn(0x7f1aa806eb70 msgr2=0x7f1aa8070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.934+0000 7f1aa7fff700 1 --2- 172.21.15.113:0/4057905231 >> v2:172.21.15.141:3300/0 conn(0x7f1aa806eb70 0x7f1aa8070bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.934+0000 7f1aa7fff700 1 -- 172.21.15.113:0/4057905231 >> v1:172.21.15.141:6789/0 conn(0x7f1aa8158770 legacy=0x7f1aa806e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.934+0000 7f1aa7fff700 1 -- 172.21.15.113:0/4057905231 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa80b38a0 con 0x7f1aa815b140 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.934+0000 7f1aa6ffd700 1 --2- 172.21.15.113:0/4057905231 >> v2:172.21.15.141:3300/0 conn(0x7f1aa806eb70 0x7f1aa8070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.934+0000 7f1aa7fff700 1 --2- 172.21.15.113:0/4057905231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aa815b140 0x7f1aa8071100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1aa806c970 tx=0x7f1aa000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=415ad6c0e027dec7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.934+0000 7f1aa67fc700 1 -- 172.21.15.113:0/4057905231 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa000e070 con 0x7f1aa815b140 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.934+0000 7f1aa67fc700 1 -- 172.21.15.113:0/4057905231 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa0003a20 con 0x7f1aa815b140 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.934+0000 7f1aa67fc700 1 -- 172.21.15.113:0/4057905231 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0003dd0 con 0x7f1aa815b140 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.935+0000 7f1aadc68700 1 -- 172.21.15.113:0/4057905231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aa815b140 msgr2=0x7f1aa8071100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.935+0000 7f1aadc68700 1 --2- 172.21.15.113:0/4057905231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aa815b140 0x7f1aa8071100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1aa806c970 tx=0x7f1aa000b330 comp rx=0 tx=0).stop 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.935+0000 7f1aadc68700 1 -- 172.21.15.113:0/4057905231 shutdown_connections 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.935+0000 7f1aadc68700 1 --2- 172.21.15.113:0/4057905231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aa815b140 0x7f1aa8071100 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.935+0000 7f1aadc68700 1 --2- 172.21.15.113:0/4057905231 >> v2:172.21.15.141:3300/0 conn(0x7f1aa806eb70 0x7f1aa8070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.935+0000 7f1aadc68700 1 -- 172.21.15.113:0/4057905231 >> 172.21.15.113:0/4057905231 conn(0x7f1aa80b1740 msgr2=0x7f1aa80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:50.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.935+0000 7f1aadc68700 1 -- 172.21.15.113:0/4057905231 shutdown_connections 2024-01-15T11:25:50.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.935+0000 7f1aadc68700 1 -- 172.21.15.113:0/4057905231 wait complete. 2024-01-15T11:25:50.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.936+0000 7f1aadc68700 1 Processor -- start 2024-01-15T11:25:50.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.936+0000 7f1aadc68700 1 -- start start 2024-01-15T11:25:50.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aadc68700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1aa806eb70 0x7f1aa80cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aadc68700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aa8158770 0x7f1aa80ce740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aadc68700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1aa815b140 0x7f1aa814e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aadc68700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa8152e30 con 0x7f1aa8158770 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aadc68700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa815a320 con 0x7f1aa815b140 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aadc68700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa80cedb0 con 0x7f1aa806eb70 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aa6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aa8158770 0x7f1aa80ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aa6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aa8158770 0x7f1aa80ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44948/0 (socket says 172.21.15.113:44948) 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aa6ffd700 1 -- 172.21.15.113:0/3139241329 learned_addr learned my addr 172.21.15.113:0/3139241329 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aa7fff700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1aa815b140 0x7f1aa814e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aa77fe700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1aa806eb70 0x7f1aa80cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aa6ffd700 1 -- 172.21.15.113:0/3139241329 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1aa806eb70 msgr2=0x7f1aa80cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aa6ffd700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1aa806eb70 0x7f1aa80cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aa6ffd700 1 -- 172.21.15.113:0/3139241329 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1aa815b140 msgr2=0x7f1aa814e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aa6ffd700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1aa815b140 0x7f1aa814e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aa6ffd700 1 -- 172.21.15.113:0/3139241329 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa0009b30 con 0x7f1aa8158770 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1aa6ffd700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aa8158770 0x7f1aa80ce740 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000c820 tx=0x7f1a9000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1a9ffff700 1 -- 172.21.15.113:0/3139241329 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90011800 con 0x7f1aa8158770 2024-01-15T11:25:50.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1a9ffff700 1 -- 172.21.15.113:0/3139241329 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a90011960 con 0x7f1aa8158770 2024-01-15T11:25:50.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.937+0000 7f1a9ffff700 1 -- 172.21.15.113:0/3139241329 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90011d10 con 0x7f1aa8158770 2024-01-15T11:25:50.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.938+0000 7f1aadc68700 1 -- 172.21.15.113:0/3139241329 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa8153080 con 0x7f1aa8158770 2024-01-15T11:25:50.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.938+0000 7f1aadc68700 1 -- 172.21.15.113:0/3139241329 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa814ea20 con 0x7f1aa8158770 2024-01-15T11:25:50.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.938+0000 7f1aadc68700 1 -- 172.21.15.113:0/3139241329 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa8153880 con 0x7f1aa8158770 2024-01-15T11:25:50.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.939+0000 7f1a9ffff700 1 -- 172.21.15.113:0/3139241329 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1a90018360 con 0x7f1aa8158770 2024-01-15T11:25:50.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.939+0000 7f1a9ffff700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f1a8806b350 0x7f1a8806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:50.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.939+0000 7f1a9ffff700 1 -- 172.21.15.113:0/3139241329 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a90089ba0 con 0x7f1aa8158770 2024-01-15T11:25:50.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.939+0000 7f1aa77fe700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f1a8806b350 0x7f1a8806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:50.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.940+0000 7f1aa77fe700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f1a8806b350 0x7f1a8806d810 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1a98004f30 tx=0x7f1a98009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:50.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:50.944+0000 7f1a9ffff700 1 -- 172.21.15.113:0/3139241329 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1aa8153880 con 0x7f1aa8158770 2024-01-15T11:25:51.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.134+0000 7f1aadc68700 1 -- 172.21.15.113:0/3139241329 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1aa8153880 con 0x7f1aa8158770 2024-01-15T11:25:51.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.135+0000 7f1a9ffff700 1 -- 172.21.15.113:0/3139241329 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f1aa8153880 con 0x7f1aa8158770 2024-01-15T11:25:51.134 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:51.135 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:51.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.138+0000 7f1aadc68700 1 -- 172.21.15.113:0/3139241329 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f1a8806b350 msgr2=0x7f1a8806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.138+0000 7f1aadc68700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f1a8806b350 0x7f1a8806d810 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1a98004f30 tx=0x7f1a98009040 comp rx=0 tx=0).stop 2024-01-15T11:25:51.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.138+0000 7f1aadc68700 1 -- 172.21.15.113:0/3139241329 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aa8158770 msgr2=0x7f1aa80ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.138+0000 7f1aadc68700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aa8158770 0x7f1aa80ce740 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000c820 tx=0x7f1a9000cbe0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.138+0000 7f1aadc68700 1 -- 172.21.15.113:0/3139241329 shutdown_connections 2024-01-15T11:25:51.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.138+0000 7f1aadc68700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f1a8806b350 0x7f1a8806d810 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.138+0000 7f1aadc68700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1aa815b140 0x7f1aa814e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.138+0000 7f1aadc68700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f1aa806eb70 0x7f1aa80cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.138+0000 7f1aadc68700 1 --2- 172.21.15.113:0/3139241329 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aa8158770 0x7f1aa80ce740 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.138+0000 7f1aadc68700 1 -- 172.21.15.113:0/3139241329 >> 172.21.15.113:0/3139241329 conn(0x7f1aa80b1740 msgr2=0x7f1aa8071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:51.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.138+0000 7f1aadc68700 1 -- 172.21.15.113:0/3139241329 shutdown_connections 2024-01-15T11:25:51.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.138+0000 7f1aadc68700 1 -- 172.21.15.113:0/3139241329 wait complete. 2024-01-15T11:25:51.149 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T11:25:13.398220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T11:25:14.589439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:51.149 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:51.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.274+0000 7fb6ea5ad700 1 Processor -- start 2024-01-15T11:25:51.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.274+0000 7fb6ea5ad700 1 -- start start 2024-01-15T11:25:51.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.275+0000 7fb6ea5ad700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb6e415e0c0 0x7fb6e40afb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.275+0000 7fb6ea5ad700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e415e4a0 con 0x7fb6e40b3bf0 2024-01-15T11:25:51.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.275+0000 7fb6ea5ad700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e406aff0 con 0x7fb6e40746a0 2024-01-15T11:25:51.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.275+0000 7fb6ea5ad700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e406b2d0 con 0x7fb6e415e0c0 2024-01-15T11:25:51.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.275+0000 7fb6e37fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb6e40b3bf0 0x7fb6e4073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42284/0 (socket says 172.21.15.113:42284) 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.275+0000 7fb6e37fe700 1 -- 172.21.15.113:0/1639323881 learned_addr learned my addr 172.21.15.113:0/1639323881 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325205244 0 0) 0x7fb6e415e4a0 con 0x7fb6e40b3bf0 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d00035b0 con 0x7fb6e40b3bf0 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e3fff700 1 --2- 172.21.15.113:0/1639323881 >> v2:172.21.15.141:3300/0 conn(0x7fb6e415e0c0 0x7fb6e40afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3079789440 0 0) 0x7fb6d00035b0 con 0x7fb6e40b3bf0 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 >> v2:172.21.15.141:3300/0 conn(0x7fb6e415e0c0 msgr2=0x7fb6e40afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 --2- 172.21.15.113:0/1639323881 >> v2:172.21.15.141:3300/0 conn(0x7fb6e415e0c0 0x7fb6e40afb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 >> v1:172.21.15.141:6789/0 conn(0x7fb6e40746a0 legacy=0x7fb6e40add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e406b5b0 con 0x7fb6e40b3bf0 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fb6d4001670 con 0x7fb6e40b3bf0 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 >> v1:172.21.15.113:6789/0 conn(0x7fb6e40b3bf0 legacy=0x7fb6e4073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 --2- 172.21.15.113:0/1639323881 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6d0003a50 0x7fb6d0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 --2- 172.21.15.113:0/1639323881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 0x7fb6d0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 --2- 172.21.15.113:0/1639323881 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6d0005330 0x7fb6d000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e406b2d0 con 0x7fb6d0004850 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d000c250 con 0x7fb6d0005330 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d000c410 con 0x7fb6d0003a50 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e37fe700 1 --2- 172.21.15.113:0/1639323881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 0x7fb6d0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e3fff700 1 --2- 172.21.15.113:0/1639323881 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6d0003a50 0x7fb6d0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e8b4a700 1 --2- 172.21.15.113:0/1639323881 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6d0005330 0x7fb6d000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e37fe700 1 -- 172.21.15.113:0/1639323881 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6d0003a50 msgr2=0x7fb6d0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e37fe700 1 --2- 172.21.15.113:0/1639323881 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6d0003a50 0x7fb6d0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e37fe700 1 -- 172.21.15.113:0/1639323881 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6d0005330 msgr2=0x7fb6d000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e37fe700 1 --2- 172.21.15.113:0/1639323881 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6d0005330 0x7fb6d000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e8b4a700 1 -- 172.21.15.113:0/1639323881 reap_dead start 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e37fe700 1 -- 172.21.15.113:0/1639323881 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d00035b0 con 0x7fb6d0004850 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.276+0000 7fb6e37fe700 1 --2- 172.21.15.113:0/1639323881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 0x7fb6d0004cd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400c110 tx=0x7fb6d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41d2b925dbe0a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.277+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d400ccb0 con 0x7fb6d0004850 2024-01-15T11:25:51.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.277+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d400ce30 con 0x7fb6d0004850 2024-01-15T11:25:51.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.277+0000 7fb6e2ffd700 1 -- 172.21.15.113:0/1639323881 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4005ba0 con 0x7fb6d0004850 2024-01-15T11:25:51.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.277+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/1639323881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 msgr2=0x7fb6d0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.277+0000 7fb6ea5ad700 1 --2- 172.21.15.113:0/1639323881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 0x7fb6d0004cd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400c110 tx=0x7fb6d0005880 comp rx=0 tx=0).stop 2024-01-15T11:25:51.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.277+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/1639323881 shutdown_connections 2024-01-15T11:25:51.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.277+0000 7fb6ea5ad700 1 --2- 172.21.15.113:0/1639323881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 0x7fb6d0004cd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.277+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/1639323881 >> 172.21.15.113:0/1639323881 conn(0x7fb6e40d0c00 msgr2=0x7fb6e40d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:51.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.277+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/1639323881 shutdown_connections 2024-01-15T11:25:51.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.277+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/1639323881 wait complete. 2024-01-15T11:25:51.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.278+0000 7fb6ea5ad700 1 Processor -- start 2024-01-15T11:25:51.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.279+0000 7fb6ea5ad700 1 -- start start 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.279+0000 7fb6ea5ad700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 0x7fb6e415f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.279+0000 7fb6ea5ad700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e40cf980 0x7fb6e41f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.279+0000 7fb6ea5ad700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e4152fc0 0x7fb6e4153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.279+0000 7fb6ea5ad700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e40ccf80 con 0x7fb6d0004850 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.279+0000 7fb6ea5ad700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e40cd240 con 0x7fb6e4152fc0 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.279+0000 7fb6ea5ad700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e40cd520 con 0x7fb6e40cf980 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.279+0000 7fb6e3fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 0x7fb6e415f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.279+0000 7fb6e3fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 0x7fb6e415f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44976/0 (socket says 172.21.15.113:44976) 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.279+0000 7fb6e3fff700 1 -- 172.21.15.113:0/4285905035 learned_addr learned my addr 172.21.15.113:0/4285905035 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6e8b4a700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e4152fc0 0x7fb6e4153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6e37fe700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e40cf980 0x7fb6e41f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6e3fff700 1 -- 172.21.15.113:0/4285905035 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e40cf980 msgr2=0x7fb6e41f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6e3fff700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e40cf980 0x7fb6e41f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6e3fff700 1 -- 172.21.15.113:0/4285905035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e4152fc0 msgr2=0x7fb6e4153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6e3fff700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e4152fc0 0x7fb6e4153450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6e3fff700 1 -- 172.21.15.113:0/4285905035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e40cd800 con 0x7fb6d0004850 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6e8b4a700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e4152fc0 0x7fb6e4153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6e3fff700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 0x7fb6e415f530 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc011900 tx=0x7fb6cc011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6e0ff9700 1 -- 172.21.15.113:0/4285905035 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc00c070 con 0x7fb6d0004850 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6e0ff9700 1 -- 172.21.15.113:0/4285905035 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6cc00f8d0 con 0x7fb6d0004850 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/4285905035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e415fa70 con 0x7fb6d0004850 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6e0ff9700 1 -- 172.21.15.113:0/4285905035 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc00fc80 con 0x7fb6d0004850 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.280+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/4285905035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e4160840 con 0x7fb6d0004850 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.281+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/4285905035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e40ccd30 con 0x7fb6d0004850 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.281+0000 7fb6e0ff9700 1 -- 172.21.15.113:0/4285905035 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb6cc015460 con 0x7fb6d0004850 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.282+0000 7fb6e0ff9700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb6c406b350 0x7fb6c406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.282+0000 7fb6e37fe700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb6c406b350 0x7fb6c406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.282+0000 7fb6e0ff9700 1 -- 172.21.15.113:0/4285905035 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6cc089ef0 con 0x7fb6d0004850 2024-01-15T11:25:51.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.282+0000 7fb6e37fe700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb6c406b350 0x7fb6c406d810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4015040 tx=0x7fb6d400a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:51.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.286+0000 7fb6e0ff9700 1 -- 172.21.15.113:0/4285905035 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6cc054d60 con 0x7fb6d0004850 2024-01-15T11:25:51.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.475+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/4285905035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb6e40cd990 con 0x7fb6d0004850 2024-01-15T11:25:51.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.475+0000 7fb6e0ff9700 1 -- 172.21.15.113:0/4285905035 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb6cc023020 con 0x7fb6d0004850 2024-01-15T11:25:51.474 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:51.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.478+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/4285905035 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb6c406b350 msgr2=0x7fb6c406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.478+0000 7fb6ea5ad700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb6c406b350 0x7fb6c406d810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4015040 tx=0x7fb6d400a000 comp rx=0 tx=0).stop 2024-01-15T11:25:51.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.478+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/4285905035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 msgr2=0x7fb6e415f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.478+0000 7fb6ea5ad700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 0x7fb6e415f530 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc011900 tx=0x7fb6cc011cc0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.478+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/4285905035 shutdown_connections 2024-01-15T11:25:51.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.478+0000 7fb6ea5ad700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb6c406b350 0x7fb6c406d810 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.478+0000 7fb6ea5ad700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e4152fc0 0x7fb6e4153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.478+0000 7fb6ea5ad700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d0004850 0x7fb6e415f530 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.478+0000 7fb6ea5ad700 1 --2- 172.21.15.113:0/4285905035 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e40cf980 0x7fb6e41f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.478+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/4285905035 >> 172.21.15.113:0/4285905035 conn(0x7fb6e40d0c00 msgr2=0x7fb6e40b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:51.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.478+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/4285905035 shutdown_connections 2024-01-15T11:25:51.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.478+0000 7fb6ea5ad700 1 -- 172.21.15.113:0/4285905035 wait complete. 2024-01-15T11:25:51.488 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T11:25:51.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.614+0000 7f46e2fee700 1 Processor -- start 2024-01-15T11:25:51.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.615+0000 7f46e2fee700 1 -- start start 2024-01-15T11:25:51.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.615+0000 7f46e2fee700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46dc15e480 0x7f46dc06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.615+0000 7f46e2fee700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46dc06aff0 0x7f46dc06b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.615+0000 7f46e2fee700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc15e860 con 0x7f46dc06c550 2024-01-15T11:25:51.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.615+0000 7f46e2fee700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46dc0cf490 con 0x7f46dc06aff0 2024-01-15T11:25:51.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.615+0000 7f46e2fee700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc0701c0 con 0x7f46dc15e480 2024-01-15T11:25:51.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46e0d8a700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46dc15e480 0x7f46dc06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46e158b700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46dc06aff0 0x7f46dc06b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46dbfff700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f46dc06c550 0x7f46dc06f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:55214/0 (socket says 172.21.15.113:55214) 2024-01-15T11:25:51.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46e158b700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f46dc06aff0 0x7f46dc06b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:50338/0 (socket says 172.21.15.113:50338) 2024-01-15T11:25:51.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46dbfff700 1 -- 172.21.15.113:0/3845732760 learned_addr learned my addr 172.21.15.113:0/3845732760 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:51.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46e0d8a700 1 --2- 172.21.15.113:0/3845732760 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46dc15e480 0x7f46dc06ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:49872/0 (socket says 172.21.15.113:49872) 2024-01-15T11:25:51.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46e0d8a700 1 -- 172.21.15.113:0/3845732760 >> v2:172.21.15.141:3300/0 conn(0x7f46dc06aff0 msgr2=0x7f46dc06b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:51.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46e0d8a700 1 --2- 172.21.15.113:0/3845732760 >> v2:172.21.15.141:3300/0 conn(0x7f46dc06aff0 0x7f46dc06b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46e0d8a700 1 -- 172.21.15.113:0/3845732760 >> v1:172.21.15.141:6789/0 conn(0x7f46dc06c550 legacy=0x7f46dc06f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46e0d8a700 1 -- 172.21.15.113:0/3845732760 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc06cb40 con 0x7f46dc15e480 2024-01-15T11:25:51.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46e158b700 1 --2- 172.21.15.113:0/3845732760 >> v2:172.21.15.141:3300/0 conn(0x7f46dc06aff0 0x7f46dc06b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:51.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46e0d8a700 1 --2- 172.21.15.113:0/3845732760 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46dc15e480 0x7f46dc06ee80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009840 tx=0x7f46d0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=825c2819d72af521 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:51.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46db7fe700 1 -- 172.21.15.113:0/3845732760 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0011830 con 0x7f46dc15e480 2024-01-15T11:25:51.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46db7fe700 1 -- 172.21.15.113:0/3845732760 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d0011990 con 0x7f46dc15e480 2024-01-15T11:25:51.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.616+0000 7f46db7fe700 1 -- 172.21.15.113:0/3845732760 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0011d20 con 0x7f46dc15e480 2024-01-15T11:25:51.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.617+0000 7f46e2fee700 1 -- 172.21.15.113:0/3845732760 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46dc15e480 msgr2=0x7f46dc06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.617+0000 7f46e2fee700 1 --2- 172.21.15.113:0/3845732760 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46dc15e480 0x7f46dc06ee80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009840 tx=0x7f46d0009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:51.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.617+0000 7f46e2fee700 1 -- 172.21.15.113:0/3845732760 shutdown_connections 2024-01-15T11:25:51.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.617+0000 7f46e2fee700 1 --2- 172.21.15.113:0/3845732760 >> v2:172.21.15.141:3300/0 conn(0x7f46dc06aff0 0x7f46dc06b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.617+0000 7f46e2fee700 1 --2- 172.21.15.113:0/3845732760 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46dc15e480 0x7f46dc06ee80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.617+0000 7f46e2fee700 1 -- 172.21.15.113:0/3845732760 >> 172.21.15.113:0/3845732760 conn(0x7f46dc0b37f0 msgr2=0x7f46dc0b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:51.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.617+0000 7f46e2fee700 1 -- 172.21.15.113:0/3845732760 shutdown_connections 2024-01-15T11:25:51.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.617+0000 7f46e2fee700 1 -- 172.21.15.113:0/3845732760 wait complete. 2024-01-15T11:25:51.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.618+0000 7f46e2fee700 1 Processor -- start 2024-01-15T11:25:51.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.618+0000 7f46e2fee700 1 -- start start 2024-01-15T11:25:51.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.619+0000 7f46e2fee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46dc06aff0 0x7f46dc14ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.619+0000 7f46e2fee700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46dc06c550 0x7f46dc157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.619+0000 7f46e2fee700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46dc15e480 0x7f46dc0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.619+0000 7f46e2fee700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc155a80 con 0x7f46dc06aff0 2024-01-15T11:25:51.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.619+0000 7f46e2fee700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc14f350 con 0x7f46dc06c550 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.619+0000 7f46e2fee700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc1581e0 con 0x7f46dc15e480 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.619+0000 7f46e0d8a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46dc06aff0 0x7f46dc14ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.619+0000 7f46e0d8a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46dc06aff0 0x7f46dc14ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44980/0 (socket says 172.21.15.113:44980) 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.619+0000 7f46e0d8a700 1 -- 172.21.15.113:0/1898667288 learned_addr learned my addr 172.21.15.113:0/1898667288 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.619+0000 7f46dbfff700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46dc06c550 0x7f46dc157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.619+0000 7f46e158b700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46dc15e480 0x7f46dc0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.620+0000 7f46e0d8a700 1 -- 172.21.15.113:0/1898667288 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46dc15e480 msgr2=0x7f46dc0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.620+0000 7f46e0d8a700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46dc15e480 0x7f46dc0c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.620+0000 7f46e0d8a700 1 -- 172.21.15.113:0/1898667288 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46dc06c550 msgr2=0x7f46dc157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.620+0000 7f46e0d8a700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46dc06c550 0x7f46dc157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.620+0000 7f46e0d8a700 1 -- 172.21.15.113:0/1898667288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d4009a70 con 0x7f46dc06aff0 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.620+0000 7f46e0d8a700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46dc06aff0 0x7f46dc14ee10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f46d0000c00 tx=0x7f46d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.620+0000 7f46d97fa700 1 -- 172.21.15.113:0/1898667288 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d00036a0 con 0x7f46dc06aff0 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.620+0000 7f46d97fa700 1 -- 172.21.15.113:0/1898667288 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d001f070 con 0x7f46dc06aff0 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.620+0000 7f46e2fee700 1 -- 172.21.15.113:0/1898667288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d00094a0 con 0x7f46dc06aff0 2024-01-15T11:25:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.620+0000 7f46d97fa700 1 -- 172.21.15.113:0/1898667288 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0003a20 con 0x7f46dc06aff0 2024-01-15T11:25:51.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.620+0000 7f46e2fee700 1 -- 172.21.15.113:0/1898667288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46dc155450 con 0x7f46dc06aff0 2024-01-15T11:25:51.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.621+0000 7f46e2fee700 1 -- 172.21.15.113:0/1898667288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46dc069020 con 0x7f46dc06aff0 2024-01-15T11:25:51.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.621+0000 7f46d97fa700 1 -- 172.21.15.113:0/1898667288 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f46d0003b80 con 0x7f46dc06aff0 2024-01-15T11:25:51.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.622+0000 7f46d97fa700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f46c006b350 0x7f46c006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.622+0000 7f46d97fa700 1 -- 172.21.15.113:0/1898667288 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46d0021070 con 0x7f46dc06aff0 2024-01-15T11:25:51.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.622+0000 7f46dbfff700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f46c006b350 0x7f46c006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.622+0000 7f46dbfff700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f46c006b350 0x7f46c006d810 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f46dc15a630 tx=0x7f46c8007f10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:51.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.626+0000 7f46d97fa700 1 -- 172.21.15.113:0/1898667288 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46d00561f0 con 0x7f46dc06aff0 2024-01-15T11:25:51.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.815+0000 7f46e2fee700 1 -- 172.21.15.113:0/1898667288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f46dc0d1970 con 0x7f46dc06aff0 2024-01-15T11:25:51.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.816+0000 7f46d97fa700 1 -- 172.21.15.113:0/1898667288 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f46d0059810 con 0x7f46dc06aff0 2024-01-15T11:25:51.815 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-15T11:25:51.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.818+0000 7f46e2fee700 1 -- 172.21.15.113:0/1898667288 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f46c006b350 msgr2=0x7f46c006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.818+0000 7f46e2fee700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f46c006b350 0x7f46c006d810 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f46dc15a630 tx=0x7f46c8007f10 comp rx=0 tx=0).stop 2024-01-15T11:25:51.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.818+0000 7f46e2fee700 1 -- 172.21.15.113:0/1898667288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46dc06aff0 msgr2=0x7f46dc14ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.818+0000 7f46e2fee700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46dc06aff0 0x7f46dc14ee10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f46d0000c00 tx=0x7f46d0004750 comp rx=0 tx=0).stop 2024-01-15T11:25:51.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.818+0000 7f46e2fee700 1 -- 172.21.15.113:0/1898667288 shutdown_connections 2024-01-15T11:25:51.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.818+0000 7f46e2fee700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f46c006b350 0x7f46c006d810 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.818+0000 7f46e2fee700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f46dc15e480 0x7f46dc0c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.818+0000 7f46e2fee700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46dc06aff0 0x7f46dc14ee10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.818+0000 7f46e2fee700 1 --2- 172.21.15.113:0/1898667288 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f46dc06c550 0x7f46dc157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.818+0000 7f46e2fee700 1 -- 172.21.15.113:0/1898667288 >> 172.21.15.113:0/1898667288 conn(0x7f46dc0b37f0 msgr2=0x7f46dc15c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:51.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.818+0000 7f46e2fee700 1 -- 172.21.15.113:0/1898667288 shutdown_connections 2024-01-15T11:25:51.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.818+0000 7f46e2fee700 1 -- 172.21.15.113:0/1898667288 wait complete. 2024-01-15T11:25:51.829 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T11:25:51.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.953+0000 7f780d7fd700 1 Processor -- start 2024-01-15T11:25:51.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.953+0000 7f780d7fd700 1 -- start start 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.954+0000 7f780d7fd700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f78080b37a0 0x7f780806e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.954+0000 7f780d7fd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f78080cffa0 0x7f78080d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.954+0000 7f780d7fd700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f780815e930 con 0x7f78081444c0 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.954+0000 7f780d7fd700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78080d0950 con 0x7f78080cffa0 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.954+0000 7f780d7fd700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78081448a0 con 0x7f78080b37a0 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.954+0000 7f78077fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f78080cffa0 0x7f78080d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.954+0000 7f78077fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f78080cffa0 0x7f78080d0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44990/0 (socket says 172.21.15.113:44990) 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.954+0000 7f78077fe700 1 -- 172.21.15.113:0/4055059220 learned_addr learned my addr 172.21.15.113:0/4055059220 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.955+0000 7f78067fc700 1 --2- 172.21.15.113:0/4055059220 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f78080b37a0 0x7f780806e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.955+0000 7f78077fe700 1 -- 172.21.15.113:0/4055059220 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f78080b37a0 msgr2=0x7f780806e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.955+0000 7f78077fe700 1 --2- 172.21.15.113:0/4055059220 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f78080b37a0 0x7f780806e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.955+0000 7f78077fe700 1 -- 172.21.15.113:0/4055059220 >> v1:172.21.15.141:6789/0 conn(0x7f78081444c0 legacy=0x7f78080ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.955+0000 7f78077fe700 1 -- 172.21.15.113:0/4055059220 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78080b2d80 con 0x7f78080cffa0 2024-01-15T11:25:51.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.955+0000 7f78077fe700 1 --2- 172.21.15.113:0/4055059220 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78080cffa0 0x7f78080d0410 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7800009e50 tx=0x7f780000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e797ef79b2d55e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:51.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.955+0000 7f7805ffb700 1 -- 172.21.15.113:0/4055059220 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780000e070 con 0x7f78080cffa0 2024-01-15T11:25:51.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.955+0000 7f7805ffb700 1 -- 172.21.15.113:0/4055059220 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780000b9c0 con 0x7f78080cffa0 2024-01-15T11:25:51.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.955+0000 7f7805ffb700 1 -- 172.21.15.113:0/4055059220 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780000bd70 con 0x7f78080cffa0 2024-01-15T11:25:51.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.955+0000 7f780d7fd700 1 -- 172.21.15.113:0/4055059220 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78080cffa0 msgr2=0x7f78080d0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.955+0000 7f780d7fd700 1 --2- 172.21.15.113:0/4055059220 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78080cffa0 0x7f78080d0410 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7800009e50 tx=0x7f780000b330 comp rx=0 tx=0).stop 2024-01-15T11:25:51.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.956+0000 7f780d7fd700 1 -- 172.21.15.113:0/4055059220 shutdown_connections 2024-01-15T11:25:51.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.956+0000 7f780d7fd700 1 --2- 172.21.15.113:0/4055059220 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78080cffa0 0x7f78080d0410 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.956+0000 7f780d7fd700 1 --2- 172.21.15.113:0/4055059220 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f78080b37a0 0x7f780806e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.956+0000 7f780d7fd700 1 -- 172.21.15.113:0/4055059220 >> 172.21.15.113:0/4055059220 conn(0x7f780806b410 msgr2=0x7f780806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:51.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.956+0000 7f780d7fd700 1 -- 172.21.15.113:0/4055059220 shutdown_connections 2024-01-15T11:25:51.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.956+0000 7f780d7fd700 1 -- 172.21.15.113:0/4055059220 wait complete. 2024-01-15T11:25:51.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.957+0000 7f780d7fd700 1 Processor -- start 2024-01-15T11:25:51.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.957+0000 7f780d7fd700 1 -- start start 2024-01-15T11:25:51.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f780d7fd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78080b37a0 0x7f7808140730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f780d7fd700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f78080cffa0 0x7f780815a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f780d7fd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78081444c0 0x7f7808070b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f780d7fd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78080cdd10 con 0x7f78081444c0 2024-01-15T11:25:51.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f780d7fd700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78080cdff0 con 0x7f78080b37a0 2024-01-15T11:25:51.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f780d7fd700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78080ce2d0 con 0x7f78080cffa0 2024-01-15T11:25:51.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f78077fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78081444c0 0x7f7808070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f78077fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78081444c0 0x7f7808070b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45006/0 (socket says 172.21.15.113:45006) 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f78077fe700 1 -- 172.21.15.113:0/292877223 learned_addr learned my addr 172.21.15.113:0/292877223 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f7806ffd700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78080b37a0 0x7f7808140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f78067fc700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f78080cffa0 0x7f780815a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f78077fe700 1 -- 172.21.15.113:0/292877223 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f78080cffa0 msgr2=0x7f780815a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f78077fe700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f78080cffa0 0x7f780815a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f78077fe700 1 -- 172.21.15.113:0/292877223 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78080b37a0 msgr2=0x7f7808140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f78077fe700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78080b37a0 0x7f7808140730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f78077fe700 1 -- 172.21.15.113:0/292877223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7800009b30 con 0x7f78081444c0 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f78067fc700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f78080cffa0 0x7f780815a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f7806ffd700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78080b37a0 0x7f7808140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f78077fe700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78081444c0 0x7f7808070b20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7808072dd0 tx=0x7f7800004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f77ef7fe700 1 -- 172.21.15.113:0/292877223 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78000099b0 con 0x7f78081444c0 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f77ef7fe700 1 -- 172.21.15.113:0/292877223 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78000092e0 con 0x7f78081444c0 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f77ef7fe700 1 -- 172.21.15.113:0/292877223 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780001c640 con 0x7f78081444c0 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f780d7fd700 1 -- 172.21.15.113:0/292877223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78080ce5b0 con 0x7f78081444c0 2024-01-15T11:25:51.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.958+0000 7f780d7fd700 1 -- 172.21.15.113:0/292877223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f780815a9c0 con 0x7f78081444c0 2024-01-15T11:25:51.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.959+0000 7f780d7fd700 1 -- 172.21.15.113:0/292877223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78080ad870 con 0x7f78081444c0 2024-01-15T11:25:51.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.960+0000 7f77ef7fe700 1 -- 172.21.15.113:0/292877223 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f780001c7a0 con 0x7f78081444c0 2024-01-15T11:25:51.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.960+0000 7f77ef7fe700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f77f406b350 0x7f77f406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:51.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.960+0000 7f77ef7fe700 1 -- 172.21.15.113:0/292877223 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f780008b820 con 0x7f78081444c0 2024-01-15T11:25:51.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.960+0000 7f7806ffd700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f77f406b350 0x7f77f406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:51.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.961+0000 7f7806ffd700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f77f406b350 0x7f77f406d810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f77fc000f60 tx=0x7f77fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:51.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:51.964+0000 7f77ef7fe700 1 -- 172.21.15.113:0/292877223 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78080ad870 con 0x7f78081444c0 2024-01-15T11:25:52.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.155+0000 7f780d7fd700 1 -- 172.21.15.113:0/292877223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f78080ad870 con 0x7f78081444c0 2024-01-15T11:25:52.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.156+0000 7f77ef7fe700 1 -- 172.21.15.113:0/292877223 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f78080ad870 con 0x7f78081444c0 2024-01-15T11:25:52.155 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-15T11:25:52.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.158+0000 7f780d7fd700 1 -- 172.21.15.113:0/292877223 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f77f406b350 msgr2=0x7f77f406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.158+0000 7f780d7fd700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f77f406b350 0x7f77f406d810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f77fc000f60 tx=0x7f77fc009040 comp rx=0 tx=0).stop 2024-01-15T11:25:52.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.158+0000 7f780d7fd700 1 -- 172.21.15.113:0/292877223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78081444c0 msgr2=0x7f7808070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.158+0000 7f780d7fd700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78081444c0 0x7f7808070b20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7808072dd0 tx=0x7f7800004ad0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.158+0000 7f780d7fd700 1 -- 172.21.15.113:0/292877223 shutdown_connections 2024-01-15T11:25:52.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.158+0000 7f780d7fd700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f77f406b350 0x7f77f406d810 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.158+0000 7f780d7fd700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78081444c0 0x7f7808070b20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.158+0000 7f780d7fd700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f78080b37a0 0x7f7808140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.158+0000 7f780d7fd700 1 --2- 172.21.15.113:0/292877223 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f78080cffa0 0x7f780815a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.158+0000 7f780d7fd700 1 -- 172.21.15.113:0/292877223 >> 172.21.15.113:0/292877223 conn(0x7f780806b410 msgr2=0x7f7808072460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:52.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.158+0000 7f780d7fd700 1 -- 172.21.15.113:0/292877223 shutdown_connections 2024-01-15T11:25:52.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.158+0000 7f780d7fd700 1 -- 172.21.15.113:0/292877223 wait complete. 2024-01-15T11:25:52.169 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T11:25:52.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.293+0000 7f2728e89700 1 Processor -- start 2024-01-15T11:25:52.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.293+0000 7f2728e89700 1 -- start start 2024-01-15T11:25:52.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f2728e89700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f272415c140 0x7f27240d02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f2728e89700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2724074e90 con 0x7f272406ee50 2024-01-15T11:25:52.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f2728e89700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272415c700 con 0x7f2724073f90 2024-01-15T11:25:52.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f2728e89700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f272415c9e0 con 0x7f272415c140 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f2721d9b700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f272406ee50 0x7f2724074370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42296/0 (socket says 172.21.15.113:42296) 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f2721d9b700 1 -- 172.21.15.113:0/1539791577 learned_addr learned my addr 172.21.15.113:0/1539791577 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264954441 0 0) 0x7f2724074e90 con 0x7f272406ee50 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f270c0035b0 con 0x7f272406ee50 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f2722d9d700 1 --2- 172.21.15.113:0/1539791577 >> v2:172.21.15.141:3300/0 conn(0x7f272415c140 0x7f27240d02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 574030002 0 0) 0x7f270c0035b0 con 0x7f272406ee50 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 >> v2:172.21.15.141:3300/0 conn(0x7f272415c140 msgr2=0x7f27240d02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f272159a700 1 --2- 172.21.15.113:0/1539791577 >> v2:172.21.15.141:3300/0 conn(0x7f272415c140 0x7f27240d02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 >> v1:172.21.15.141:6789/0 conn(0x7f2724073f90 legacy=0x7f27240ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272415ccc0 con 0x7f272406ee50 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f2718001670 con 0x7f272406ee50 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 >> v1:172.21.15.113:6789/0 conn(0x7f272406ee50 legacy=0x7f2724074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f272159a700 1 --2- 172.21.15.113:0/1539791577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f270c003a30 0x7f270c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.294+0000 7f272159a700 1 --2- 172.21.15.113:0/1539791577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f270c004850 0x7f270c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272159a700 1 --2- 172.21.15.113:0/1539791577 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f270c005260 0x7f270c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272415c9e0 con 0x7f270c003a30 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c00c190 con 0x7f270c004850 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c00c350 con 0x7f270c005260 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272259c700 1 --2- 172.21.15.113:0/1539791577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f270c003a30 0x7f270c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f2721d9b700 1 --2- 172.21.15.113:0/1539791577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f270c004850 0x7f270c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f2722d9d700 1 --2- 172.21.15.113:0/1539791577 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f270c005260 0x7f270c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272259c700 1 -- 172.21.15.113:0/1539791577 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f270c005260 msgr2=0x7f270c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272259c700 1 --2- 172.21.15.113:0/1539791577 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f270c005260 0x7f270c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272259c700 1 -- 172.21.15.113:0/1539791577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f270c004850 msgr2=0x7f270c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272259c700 1 --2- 172.21.15.113:0/1539791577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f270c004850 0x7f270c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272259c700 1 -- 172.21.15.113:0/1539791577 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c0035b0 con 0x7f270c003a30 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f2721d9b700 1 --2- 172.21.15.113:0/1539791577 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f270c004850 0x7f270c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f2722d9d700 1 -- 172.21.15.113:0/1539791577 reap_dead start 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272259c700 1 --2- 172.21.15.113:0/1539791577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f270c003a30 0x7f270c003e50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f27140017c0 tx=0x7f270c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb9cfcc830e74fa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:52.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27140052d0 con 0x7f270c003a30 2024-01-15T11:25:52.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2714005430 con 0x7f270c003a30 2024-01-15T11:25:52.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.295+0000 7f272159a700 1 -- 172.21.15.113:0/1539791577 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27140038c0 con 0x7f270c003a30 2024-01-15T11:25:52.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.296+0000 7f2728e89700 1 -- 172.21.15.113:0/1539791577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f270c003a30 msgr2=0x7f270c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.296+0000 7f2728e89700 1 --2- 172.21.15.113:0/1539791577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f270c003a30 0x7f270c003e50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f27140017c0 tx=0x7f270c005880 comp rx=0 tx=0).stop 2024-01-15T11:25:52.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.296+0000 7f2728e89700 1 -- 172.21.15.113:0/1539791577 shutdown_connections 2024-01-15T11:25:52.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.296+0000 7f2728e89700 1 --2- 172.21.15.113:0/1539791577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f270c003a30 0x7f270c003e50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.296+0000 7f2728e89700 1 -- 172.21.15.113:0/1539791577 >> 172.21.15.113:0/1539791577 conn(0x7f27240cd570 msgr2=0x7f27240cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:52.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.296+0000 7f2728e89700 1 -- 172.21.15.113:0/1539791577 shutdown_connections 2024-01-15T11:25:52.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.296+0000 7f2728e89700 1 -- 172.21.15.113:0/1539791577 wait complete. 2024-01-15T11:25:52.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.297+0000 7f2728e89700 1 Processor -- start 2024-01-15T11:25:52.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.297+0000 7f2728e89700 1 -- start start 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2728e89700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f270c003a30 0x7f2724150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2728e89700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27240cb780 0x7f27240cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2728e89700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240c9e60 0x7f27240ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2728e89700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27240cc140 con 0x7f27240c9e60 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2728e89700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2724150ba0 con 0x7f27240cb780 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2728e89700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27240ca850 con 0x7f270c003a30 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2722d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240c9e60 0x7f27240ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2722d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240c9e60 0x7f27240ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45028/0 (socket says 172.21.15.113:45028) 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2722d9d700 1 -- 172.21.15.113:0/2305986856 learned_addr learned my addr 172.21.15.113:0/2305986856 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2722d9d700 1 -- 172.21.15.113:0/2305986856 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f270c003a30 msgr2=0x7f2724150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2722d9d700 1 --2- 172.21.15.113:0/2305986856 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f270c003a30 0x7f2724150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2722d9d700 1 -- 172.21.15.113:0/2305986856 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27240cb780 msgr2=0x7f27240cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2722d9d700 1 --2- 172.21.15.113:0/2305986856 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27240cb780 0x7f27240cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2722d9d700 1 -- 172.21.15.113:0/2305986856 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27240cab20 con 0x7f27240c9e60 2024-01-15T11:25:52.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2722d9d700 1 --2- 172.21.15.113:0/2305986856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240c9e60 0x7f27240ca310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f272415c520 tx=0x7f271c0114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:52.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.298+0000 7f2712ffd700 1 -- 172.21.15.113:0/2305986856 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c00e070 con 0x7f27240c9e60 2024-01-15T11:25:52.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.299+0000 7f2712ffd700 1 -- 172.21.15.113:0/2305986856 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271c003c30 con 0x7f27240c9e60 2024-01-15T11:25:52.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.299+0000 7f2728e89700 1 -- 172.21.15.113:0/2305986856 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27240cae00 con 0x7f27240c9e60 2024-01-15T11:25:52.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.299+0000 7f2712ffd700 1 -- 172.21.15.113:0/2305986856 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c003e20 con 0x7f27240c9e60 2024-01-15T11:25:52.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.299+0000 7f2728e89700 1 -- 172.21.15.113:0/2305986856 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27240cb3c0 con 0x7f27240c9e60 2024-01-15T11:25:52.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.300+0000 7f2728e89700 1 -- 172.21.15.113:0/2305986856 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2724067f60 con 0x7f27240c9e60 2024-01-15T11:25:52.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.300+0000 7f2712ffd700 1 -- 172.21.15.113:0/2305986856 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f271c011a10 con 0x7f27240c9e60 2024-01-15T11:25:52.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.300+0000 7f2712ffd700 1 --2- 172.21.15.113:0/2305986856 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f270406b350 0x7f270406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.301+0000 7f272259c700 1 --2- 172.21.15.113:0/2305986856 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f270406b350 0x7f270406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:52.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.301+0000 7f2712ffd700 1 -- 172.21.15.113:0/2305986856 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f271c089f90 con 0x7f27240c9e60 2024-01-15T11:25:52.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.301+0000 7f272259c700 1 --2- 172.21.15.113:0/2305986856 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f270406b350 0x7f270406d810 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2714005040 tx=0x7f2714011000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:52.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.305+0000 7f2712ffd700 1 -- 172.21.15.113:0/2305986856 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f271c054e00 con 0x7f27240c9e60 2024-01-15T11:25:52.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.513+0000 7f2728e89700 1 -- 172.21.15.113:0/2305986856 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f27240d0860 con 0x7f27240c9e60 2024-01-15T11:25:52.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.513+0000 7f2712ffd700 1 -- 172.21.15.113:0/2305986856 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f271c058420 con 0x7f27240c9e60 2024-01-15T11:25:52.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.515+0000 7f2728e89700 1 -- 172.21.15.113:0/2305986856 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f270406b350 msgr2=0x7f270406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.515+0000 7f2728e89700 1 --2- 172.21.15.113:0/2305986856 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f270406b350 0x7f270406d810 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2714005040 tx=0x7f2714011000 comp rx=0 tx=0).stop 2024-01-15T11:25:52.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.515+0000 7f2728e89700 1 -- 172.21.15.113:0/2305986856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240c9e60 msgr2=0x7f27240ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.515+0000 7f2728e89700 1 --2- 172.21.15.113:0/2305986856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240c9e60 0x7f27240ca310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f272415c520 tx=0x7f271c0114b0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.515+0000 7f2728e89700 1 -- 172.21.15.113:0/2305986856 shutdown_connections 2024-01-15T11:25:52.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.515+0000 7f2728e89700 1 --2- 172.21.15.113:0/2305986856 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f270406b350 0x7f270406d810 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.515+0000 7f2728e89700 1 --2- 172.21.15.113:0/2305986856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240c9e60 0x7f27240ca310 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.515+0000 7f2728e89700 1 --2- 172.21.15.113:0/2305986856 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f270c003a30 0x7f2724150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.515+0000 7f2728e89700 1 --2- 172.21.15.113:0/2305986856 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f27240cb780 0x7f27240cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.516+0000 7f2728e89700 1 -- 172.21.15.113:0/2305986856 >> 172.21.15.113:0/2305986856 conn(0x7f27240cd570 msgr2=0x7f27240b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:52.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.516+0000 7f2728e89700 1 -- 172.21.15.113:0/2305986856 shutdown_connections 2024-01-15T11:25:52.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.516+0000 7f2728e89700 1 -- 172.21.15.113:0/2305986856 wait complete. 2024-01-15T11:25:52.516 INFO:teuthology.orchestra.run.smithi113.stderr:listed 0 entries 2024-01-15T11:25:52.527 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-01-15T11:25:52.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:52.527 DEBUG:teuthology.orchestra.run.smithi141:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-01-15T11:25:52.578 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-15T11:25:52.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:52.578 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T11:25:52.617 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:25:52.617 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:52.636 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:25:52.636 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:52.695 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:25:52.695 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:52.753 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:52.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.914+0000 7f6aa3e22700 1 Processor -- start 2024-01-15T11:25:52.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.914+0000 7f6aa3e22700 1 -- start start 2024-01-15T11:25:52.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.915+0000 7f6aa3e22700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f6a9c1446a0 0x7f6a9c0b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.915+0000 7f6aa3e22700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 0x7f6a9c0b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.915+0000 7f6aa3e22700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9c0af950 con 0x7f6a9c0adf00 2024-01-15T11:25:52.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.915+0000 7f6aa3e22700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9c144a80 con 0x7f6a9c1446a0 2024-01-15T11:25:52.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.915+0000 7f6aa3e22700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c0afc30 con 0x7f6a9c06c710 2024-01-15T11:25:52.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.916+0000 7f6aa23bf700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 0x7f6a9c0b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:52.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.916+0000 7f6aa23bf700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 0x7f6a9c0b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:49932/0 (socket says 172.21.15.113:49932) 2024-01-15T11:25:52.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.916+0000 7f6aa23bf700 1 -- 172.21.15.113:0/2846389300 learned_addr learned my addr 172.21.15.113:0/2846389300 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:52.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.916+0000 7f6aa1bbe700 1 --2- 172.21.15.113:0/2846389300 >> v2:172.21.15.141:3300/0 conn(0x7f6a9c1446a0 0x7f6a9c0b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:52.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.916+0000 7f6aa0bbc700 1 -- 172.21.15.113:0/2846389300 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751432199 0 0) 0x7f6a9c0af950 con 0x7f6a9c0adf00 2024-01-15T11:25:52.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.916+0000 7f6aa0bbc700 1 -- 172.21.15.113:0/2846389300 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a880035b0 con 0x7f6a9c0adf00 2024-01-15T11:25:52.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.916+0000 7f6aa1bbe700 1 -- 172.21.15.113:0/2846389300 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 msgr2=0x7f6a9c0b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:52.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.916+0000 7f6aa1bbe700 1 --2- 172.21.15.113:0/2846389300 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 0x7f6a9c0b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.916+0000 7f6aa1bbe700 1 -- 172.21.15.113:0/2846389300 >> v1:172.21.15.141:6789/0 conn(0x7f6a9c0adf00 legacy=0x7f6a9c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.916+0000 7f6aa1bbe700 1 -- 172.21.15.113:0/2846389300 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9c0aff10 con 0x7f6a9c1446a0 2024-01-15T11:25:52.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.916+0000 7f6aa23bf700 1 --2- 172.21.15.113:0/2846389300 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 0x7f6a9c0b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:52.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.916+0000 7f6aa1bbe700 1 --2- 172.21.15.113:0/2846389300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a9c1446a0 0x7f6a9c0b5ae0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009840 tx=0x7f6a90009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fabee2cf294191d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:52.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.917+0000 7f6aa0bbc700 1 -- 172.21.15.113:0/2846389300 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a90011830 con 0x7f6a9c1446a0 2024-01-15T11:25:52.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.917+0000 7f6aa0bbc700 1 -- 172.21.15.113:0/2846389300 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a90011990 con 0x7f6a9c1446a0 2024-01-15T11:25:52.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.917+0000 7f6aa0bbc700 1 -- 172.21.15.113:0/2846389300 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a90011d20 con 0x7f6a9c1446a0 2024-01-15T11:25:52.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.917+0000 7f6aa3e22700 1 -- 172.21.15.113:0/2846389300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a9c1446a0 msgr2=0x7f6a9c0b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.917+0000 7f6aa3e22700 1 --2- 172.21.15.113:0/2846389300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a9c1446a0 0x7f6a9c0b5ae0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009840 tx=0x7f6a90009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:52.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.917+0000 7f6aa3e22700 1 -- 172.21.15.113:0/2846389300 shutdown_connections 2024-01-15T11:25:52.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.917+0000 7f6aa3e22700 1 --2- 172.21.15.113:0/2846389300 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 0x7f6a9c0b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.917+0000 7f6aa3e22700 1 --2- 172.21.15.113:0/2846389300 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a9c1446a0 0x7f6a9c0b5ae0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.917+0000 7f6aa3e22700 1 -- 172.21.15.113:0/2846389300 >> 172.21.15.113:0/2846389300 conn(0x7f6a9c071b90 msgr2=0x7f6a9c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:52.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.917+0000 7f6aa3e22700 1 -- 172.21.15.113:0/2846389300 shutdown_connections 2024-01-15T11:25:52.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.917+0000 7f6aa3e22700 1 -- 172.21.15.113:0/2846389300 wait complete. 2024-01-15T11:25:52.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.918+0000 7f6aa3e22700 1 Processor -- start 2024-01-15T11:25:52.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.918+0000 7f6aa3e22700 1 -- start start 2024-01-15T11:25:52.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.919+0000 7f6aa3e22700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 0x7f6a9c09aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.919+0000 7f6aa3e22700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a9c0adf00 0x7f6a9c09e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.919+0000 7f6aa3e22700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c1446a0 0x7f6a9c14edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.919+0000 7f6aa3e22700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c1506f0 con 0x7f6a9c1446a0 2024-01-15T11:25:52.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.919+0000 7f6aa3e22700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c14dd30 con 0x7f6a9c0adf00 2024-01-15T11:25:52.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.919+0000 7f6aa3e22700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c14e010 con 0x7f6a9c06c710 2024-01-15T11:25:52.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.919+0000 7f6aa23bf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c1446a0 0x7f6a9c14edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:52.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.919+0000 7f6aa23bf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c1446a0 0x7f6a9c14edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45042/0 (socket says 172.21.15.113:45042) 2024-01-15T11:25:52.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.919+0000 7f6aa23bf700 1 -- 172.21.15.113:0/3833076340 learned_addr learned my addr 172.21.15.113:0/3833076340 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:52.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6aa1bbe700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 0x7f6a9c09aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:52.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6aa13bd700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a9c0adf00 0x7f6a9c09e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:52.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6aa23bf700 1 -- 172.21.15.113:0/3833076340 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 msgr2=0x7f6a9c09aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6aa23bf700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 0x7f6a9c09aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6aa23bf700 1 -- 172.21.15.113:0/3833076340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a9c0adf00 msgr2=0x7f6a9c09e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:52.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6aa23bf700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a9c0adf00 0x7f6a9c09e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:52.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6aa23bf700 1 -- 172.21.15.113:0/3833076340 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a94009b30 con 0x7f6a9c1446a0 2024-01-15T11:25:52.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6aa13bd700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a9c0adf00 0x7f6a9c09e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:52.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6aa1bbe700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 0x7f6a9c09aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:52.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6aa23bf700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c1446a0 0x7f6a9c14edd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400efc0 tx=0x7f6a94009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:52.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6a8e7fc700 1 -- 172.21.15.113:0/3833076340 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a94018070 con 0x7f6a9c1446a0 2024-01-15T11:25:52.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6a8e7fc700 1 -- 172.21.15.113:0/3833076340 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9400f040 con 0x7f6a9c1446a0 2024-01-15T11:25:52.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6aa3e22700 1 -- 172.21.15.113:0/3833076340 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a900094a0 con 0x7f6a9c1446a0 2024-01-15T11:25:52.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6a8e7fc700 1 -- 172.21.15.113:0/3833076340 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a94003c40 con 0x7f6a9c1446a0 2024-01-15T11:25:52.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.920+0000 7f6aa3e22700 1 -- 172.21.15.113:0/3833076340 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a9c14e5d0 con 0x7f6a9c1446a0 2024-01-15T11:25:52.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.921+0000 7f6aa3e22700 1 -- 172.21.15.113:0/3833076340 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a9c0753f0 con 0x7f6a9c1446a0 2024-01-15T11:25:52.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.921+0000 7f6a8e7fc700 1 -- 172.21.15.113:0/3833076340 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6a9400c9e0 con 0x7f6a9c1446a0 2024-01-15T11:25:52.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.922+0000 7f6a8e7fc700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6a8006b350 0x7f6a8006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:52.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.922+0000 7f6a8e7fc700 1 -- 172.21.15.113:0/3833076340 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a94089e00 con 0x7f6a9c1446a0 2024-01-15T11:25:52.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.922+0000 7f6aa1bbe700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6a8006b350 0x7f6a8006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:52.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.922+0000 7f6aa1bbe700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6a8006b350 0x7f6a8006d810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6a900090b0 tx=0x7f6a90008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:52.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:52.927+0000 7f6a8e7fc700 1 -- 172.21.15.113:0/3833076340 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a94054d00 con 0x7f6a9c1446a0 2024-01-15T11:25:53.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.118+0000 7f6aa3e22700 1 -- 172.21.15.113:0/3833076340 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a9c067f60 con 0x7f6a9c1446a0 2024-01-15T11:25:53.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.119+0000 7f6a8e7fc700 1 -- 172.21.15.113:0/3833076340 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f6a94058320 con 0x7f6a9c1446a0 2024-01-15T11:25:53.118 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:53.119 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:53.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.121+0000 7f6aa3e22700 1 -- 172.21.15.113:0/3833076340 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6a8006b350 msgr2=0x7f6a8006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.121+0000 7f6aa3e22700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6a8006b350 0x7f6a8006d810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6a900090b0 tx=0x7f6a90008030 comp rx=0 tx=0).stop 2024-01-15T11:25:53.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.121+0000 7f6aa3e22700 1 -- 172.21.15.113:0/3833076340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c1446a0 msgr2=0x7f6a9c14edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.121+0000 7f6aa3e22700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c1446a0 0x7f6a9c14edd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400efc0 tx=0x7f6a94009f90 comp rx=0 tx=0).stop 2024-01-15T11:25:53.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.121+0000 7f6aa3e22700 1 -- 172.21.15.113:0/3833076340 shutdown_connections 2024-01-15T11:25:53.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.121+0000 7f6aa3e22700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f6a8006b350 0x7f6a8006d810 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.121+0000 7f6aa3e22700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c1446a0 0x7f6a9c14edd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.121+0000 7f6aa3e22700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f6a9c06c710 0x7f6a9c09aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.121+0000 7f6aa3e22700 1 --2- 172.21.15.113:0/3833076340 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a9c0adf00 0x7f6a9c09e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.121+0000 7f6aa3e22700 1 -- 172.21.15.113:0/3833076340 >> 172.21.15.113:0/3833076340 conn(0x7f6a9c071b90 msgr2=0x7f6a9c0b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:53.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.121+0000 7f6aa3e22700 1 -- 172.21.15.113:0/3833076340 shutdown_connections 2024-01-15T11:25:53.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.121+0000 7f6aa3e22700 1 -- 172.21.15.113:0/3833076340 wait complete. 2024-01-15T11:25:53.132 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T11:25:13.398220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T11:25:14.589439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:53.133 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:53.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.259+0000 7f23af06b700 1 Processor -- start 2024-01-15T11:25:53.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.259+0000 7f23af06b700 1 -- start start 2024-01-15T11:25:53.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23af06b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23a80afb70 0x7f23a815c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23af06b700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a80d01b0 con 0x7f23a815c610 2024-01-15T11:25:53.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23af06b700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a806d5b0 con 0x7f23a815bcf0 2024-01-15T11:25:53.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23af06b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a806d890 con 0x7f23a80afb70 2024-01-15T11:25:53.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23ad608700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f23a815c610 0x7f23a8071420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:42300/0 (socket says 172.21.15.113:42300) 2024-01-15T11:25:53.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23ad608700 1 -- 172.21.15.113:0/1854048755 learned_addr learned my addr 172.21.15.113:0/1854048755 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:53.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23a7fff700 1 --2- 172.21.15.113:0/1854048755 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23a80afb70 0x7f23a815c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23a77fe700 1 -- 172.21.15.113:0/1854048755 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538453148 0 0) 0x7f23a80d01b0 con 0x7f23a815c610 2024-01-15T11:25:53.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23a77fe700 1 -- 172.21.15.113:0/1854048755 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23980035b0 con 0x7f23a815c610 2024-01-15T11:25:53.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23a7fff700 1 -- 172.21.15.113:0/1854048755 >> v1:172.21.15.141:6789/0 conn(0x7f23a815bcf0 legacy=0x7f23a80cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23a7fff700 1 -- 172.21.15.113:0/1854048755 >> v1:172.21.15.113:6789/0 conn(0x7f23a815c610 legacy=0x7f23a8071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23a77fe700 1 -- 172.21.15.113:0/1854048755 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345675922 0 0) 0x7f23a806d5b0 con 0x7f23a815bcf0 2024-01-15T11:25:53.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23a7fff700 1 -- 172.21.15.113:0/1854048755 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a806db70 con 0x7f23a80afb70 2024-01-15T11:25:53.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.260+0000 7f23a7fff700 1 --2- 172.21.15.113:0/1854048755 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23a80afb70 0x7f23a815c0d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2394009870 tx=0x7f2394009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93d6889f83ef7fcb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:53.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.261+0000 7f23a77fe700 1 -- 172.21.15.113:0/1854048755 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394011830 con 0x7f23a80afb70 2024-01-15T11:25:53.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.261+0000 7f23a77fe700 1 -- 172.21.15.113:0/1854048755 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2394011990 con 0x7f23a80afb70 2024-01-15T11:25:53.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.261+0000 7f23a77fe700 1 -- 172.21.15.113:0/1854048755 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394011d20 con 0x7f23a80afb70 2024-01-15T11:25:53.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.261+0000 7f23af06b700 1 -- 172.21.15.113:0/1854048755 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23a80afb70 msgr2=0x7f23a815c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.261+0000 7f23af06b700 1 --2- 172.21.15.113:0/1854048755 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23a80afb70 0x7f23a815c0d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2394009870 tx=0x7f2394009b80 comp rx=0 tx=0).stop 2024-01-15T11:25:53.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.261+0000 7f23af06b700 1 -- 172.21.15.113:0/1854048755 shutdown_connections 2024-01-15T11:25:53.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.261+0000 7f23af06b700 1 --2- 172.21.15.113:0/1854048755 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23a80afb70 0x7f23a815c0d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.261+0000 7f23af06b700 1 -- 172.21.15.113:0/1854048755 >> 172.21.15.113:0/1854048755 conn(0x7f23a8074170 msgr2=0x7f23a806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:53.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.261+0000 7f23af06b700 1 -- 172.21.15.113:0/1854048755 shutdown_connections 2024-01-15T11:25:53.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.261+0000 7f23af06b700 1 -- 172.21.15.113:0/1854048755 wait complete. 2024-01-15T11:25:53.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.262+0000 7f23af06b700 1 Processor -- start 2024-01-15T11:25:53.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.262+0000 7f23af06b700 1 -- start start 2024-01-15T11:25:53.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.263+0000 7f23af06b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a80afb70 0x7f23a8154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.263+0000 7f23af06b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23a815bcf0 0x7f23a8142b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.263+0000 7f23af06b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23a815c610 0x7f23a8151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.263+0000 7f23af06b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a8154cb0 con 0x7f23a815bcf0 2024-01-15T11:25:53.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.263+0000 7f23af06b700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a8154f90 con 0x7f23a80afb70 2024-01-15T11:25:53.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.263+0000 7f23af06b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a8155270 con 0x7f23a815c610 2024-01-15T11:25:53.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.263+0000 7f23a7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23a815bcf0 0x7f23a8142b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.263+0000 7f23a7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23a815bcf0 0x7f23a8142b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45044/0 (socket says 172.21.15.113:45044) 2024-01-15T11:25:53.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.263+0000 7f23a7fff700 1 -- 172.21.15.113:0/4253707629 learned_addr learned my addr 172.21.15.113:0/4253707629 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:53.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23ace07700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a80afb70 0x7f23a8154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23a7fff700 1 -- 172.21.15.113:0/4253707629 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23a815c610 msgr2=0x7f23a8151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23ad608700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23a815c610 0x7f23a8151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23a7fff700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23a815c610 0x7f23a8151270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23a7fff700 1 -- 172.21.15.113:0/4253707629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a80afb70 msgr2=0x7f23a8154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23a7fff700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a80afb70 0x7f23a8154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23a7fff700 1 -- 172.21.15.113:0/4253707629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23940094a0 con 0x7f23a815bcf0 2024-01-15T11:25:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23ace07700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a80afb70 0x7f23a8154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23ad608700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23a815c610 0x7f23a8151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23a7fff700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23a815bcf0 0x7f23a8142b60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2394009bc0 tx=0x7f2394004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23a57fa700 1 -- 172.21.15.113:0/4253707629 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239401a910 con 0x7f23a815bcf0 2024-01-15T11:25:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23a57fa700 1 -- 172.21.15.113:0/4253707629 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239401aa70 con 0x7f23a815bcf0 2024-01-15T11:25:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23a57fa700 1 -- 172.21.15.113:0/4253707629 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239401ae00 con 0x7f23a815bcf0 2024-01-15T11:25:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23af06b700 1 -- 172.21.15.113:0/4253707629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a8155550 con 0x7f23a815bcf0 2024-01-15T11:25:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.264+0000 7f23af06b700 1 -- 172.21.15.113:0/4253707629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a81430a0 con 0x7f23a815bcf0 2024-01-15T11:25:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.265+0000 7f23af06b700 1 -- 172.21.15.113:0/4253707629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a814e710 con 0x7f23a815bcf0 2024-01-15T11:25:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.265+0000 7f23a57fa700 1 -- 172.21.15.113:0/4253707629 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2394022070 con 0x7f23a815bcf0 2024-01-15T11:25:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.266+0000 7f23a57fa700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f238c06b350 0x7f238c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.266+0000 7f23a57fa700 1 -- 172.21.15.113:0/4253707629 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f239408a970 con 0x7f23a815bcf0 2024-01-15T11:25:53.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.266+0000 7f23ace07700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f238c06b350 0x7f238c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.266+0000 7f23ace07700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f238c06b350 0x7f238c06d810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f239c000f60 tx=0x7f239c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:53.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.270+0000 7f23a57fa700 1 -- 172.21.15.113:0/4253707629 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2394055870 con 0x7f23a815bcf0 2024-01-15T11:25:53.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.459+0000 7f23af06b700 1 -- 172.21.15.113:0/4253707629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f23a8069020 con 0x7f23a815bcf0 2024-01-15T11:25:53.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.459+0000 7f23a57fa700 1 -- 172.21.15.113:0/4253707629 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f23940290a0 con 0x7f23a815bcf0 2024-01-15T11:25:53.458 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:53.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.461+0000 7f23af06b700 1 -- 172.21.15.113:0/4253707629 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f238c06b350 msgr2=0x7f238c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.461+0000 7f23af06b700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f238c06b350 0x7f238c06d810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f239c000f60 tx=0x7f239c009040 comp rx=0 tx=0).stop 2024-01-15T11:25:53.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.461+0000 7f23af06b700 1 -- 172.21.15.113:0/4253707629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23a815bcf0 msgr2=0x7f23a8142b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.461+0000 7f23af06b700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23a815bcf0 0x7f23a8142b60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2394009bc0 tx=0x7f2394004750 comp rx=0 tx=0).stop 2024-01-15T11:25:53.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.462+0000 7f23af06b700 1 -- 172.21.15.113:0/4253707629 shutdown_connections 2024-01-15T11:25:53.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.462+0000 7f23af06b700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f238c06b350 0x7f238c06d810 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.462+0000 7f23af06b700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23a815c610 0x7f23a8151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.462+0000 7f23af06b700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23a80afb70 0x7f23a8154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.462+0000 7f23af06b700 1 --2- 172.21.15.113:0/4253707629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23a815bcf0 0x7f23a8142b60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.462+0000 7f23af06b700 1 -- 172.21.15.113:0/4253707629 >> 172.21.15.113:0/4253707629 conn(0x7f23a8074170 msgr2=0x7f23a815f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:53.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.462+0000 7f23af06b700 1 -- 172.21.15.113:0/4253707629 shutdown_connections 2024-01-15T11:25:53.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.462+0000 7f23af06b700 1 -- 172.21.15.113:0/4253707629 wait complete. 2024-01-15T11:25:53.473 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T11:25:53.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.599+0000 7f40f898a700 1 Processor -- start 2024-01-15T11:25:53.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.599+0000 7f40f898a700 1 -- start start 2024-01-15T11:25:53.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f898a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f40f415e0c0 0x7f40f40afb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f898a700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f40f40746a0 0x7f40f40add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f898a700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f415e4a0 con 0x7f40f40b3bf0 2024-01-15T11:25:53.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f898a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f406aff0 con 0x7f40f415e0c0 2024-01-15T11:25:53.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f898a700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f406b2d0 con 0x7f40f40746a0 2024-01-15T11:25:53.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f259c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f40f415e0c0 0x7f40f40afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f259c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f40f415e0c0 0x7f40f40afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45046/0 (socket says 172.21.15.113:45046) 2024-01-15T11:25:53.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f259c700 1 -- 172.21.15.113:0/2336120026 learned_addr learned my addr 172.21.15.113:0/2336120026 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:53.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f2d9d700 1 --2- 172.21.15.113:0/2336120026 >> v2:172.21.15.141:3300/0 conn(0x7f40f40746a0 0x7f40f40add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f259c700 1 -- 172.21.15.113:0/2336120026 >> v2:172.21.15.141:3300/0 conn(0x7f40f40746a0 msgr2=0x7f40f40add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:53.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f259c700 1 --2- 172.21.15.113:0/2336120026 >> v2:172.21.15.141:3300/0 conn(0x7f40f40746a0 0x7f40f40add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f259c700 1 -- 172.21.15.113:0/2336120026 >> v1:172.21.15.141:6789/0 conn(0x7f40f40b3bf0 legacy=0x7f40f4073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f259c700 1 -- 172.21.15.113:0/2336120026 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f406b5b0 con 0x7f40f415e0c0 2024-01-15T11:25:53.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f2d9d700 1 --2- 172.21.15.113:0/2336120026 >> v2:172.21.15.141:3300/0 conn(0x7f40f40746a0 0x7f40f40add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:53.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.600+0000 7f40f259c700 1 --2- 172.21.15.113:0/2336120026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40f415e0c0 0x7f40f40afb80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f40e40098f0 tx=0x7f40e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8d1c496047d14 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:53.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.601+0000 7f40f159a700 1 -- 172.21.15.113:0/2336120026 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e400b860 con 0x7f40f415e0c0 2024-01-15T11:25:53.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.601+0000 7f40f159a700 1 -- 172.21.15.113:0/2336120026 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e400b9c0 con 0x7f40f415e0c0 2024-01-15T11:25:53.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.601+0000 7f40f159a700 1 -- 172.21.15.113:0/2336120026 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e400bd50 con 0x7f40f415e0c0 2024-01-15T11:25:53.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.601+0000 7f40f898a700 1 -- 172.21.15.113:0/2336120026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40f415e0c0 msgr2=0x7f40f40afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.601+0000 7f40f898a700 1 --2- 172.21.15.113:0/2336120026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40f415e0c0 0x7f40f40afb80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f40e40098f0 tx=0x7f40e4009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:53.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.601+0000 7f40f898a700 1 -- 172.21.15.113:0/2336120026 shutdown_connections 2024-01-15T11:25:53.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.601+0000 7f40f898a700 1 --2- 172.21.15.113:0/2336120026 >> v2:172.21.15.141:3300/0 conn(0x7f40f40746a0 0x7f40f40add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.601+0000 7f40f898a700 1 --2- 172.21.15.113:0/2336120026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40f415e0c0 0x7f40f40afb80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.601+0000 7f40f898a700 1 -- 172.21.15.113:0/2336120026 >> 172.21.15.113:0/2336120026 conn(0x7f40f40d0c00 msgr2=0x7f40f40d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:53.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.601+0000 7f40f898a700 1 -- 172.21.15.113:0/2336120026 shutdown_connections 2024-01-15T11:25:53.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.601+0000 7f40f898a700 1 -- 172.21.15.113:0/2336120026 wait complete. 2024-01-15T11:25:53.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.603+0000 7f40f898a700 1 Processor -- start 2024-01-15T11:25:53.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.603+0000 7f40f898a700 1 -- start start 2024-01-15T11:25:53.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.603+0000 7f40f898a700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f40f40746a0 0x7f40f414e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.603+0000 7f40f898a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40f40b3bf0 0x7f40f415a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.603+0000 7f40f898a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40f415e0c0 0x7f40f40cf410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.603+0000 7f40f898a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f40ce400 con 0x7f40f40b3bf0 2024-01-15T11:25:53.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.603+0000 7f40f898a700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f414edc0 con 0x7f40f415e0c0 2024-01-15T11:25:53.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.603+0000 7f40f898a700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f40ce650 con 0x7f40f40746a0 2024-01-15T11:25:53.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.603+0000 7f40f1d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40f40b3bf0 0x7f40f415a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40f1d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40f40b3bf0 0x7f40f415a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45048/0 (socket says 172.21.15.113:45048) 2024-01-15T11:25:53.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40f1d9b700 1 -- 172.21.15.113:0/1380321208 learned_addr learned my addr 172.21.15.113:0/1380321208 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:53.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40f2d9d700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40f415e0c0 0x7f40f40cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40f259c700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f40f40746a0 0x7f40f414e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40f1d9b700 1 -- 172.21.15.113:0/1380321208 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f40f40746a0 msgr2=0x7f40f414e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40f1d9b700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f40f40746a0 0x7f40f414e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40f1d9b700 1 -- 172.21.15.113:0/1380321208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40f415e0c0 msgr2=0x7f40f40cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40f1d9b700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40f415e0c0 0x7f40f40cf410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40f1d9b700 1 -- 172.21.15.113:0/1380321208 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e4009550 con 0x7f40f40b3bf0 2024-01-15T11:25:53.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40f1d9b700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40f40b3bf0 0x7f40f415a0c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f40e800c510 tx=0x7f40e800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:53.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40e2ffd700 1 -- 172.21.15.113:0/1380321208 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e80052d0 con 0x7f40f40b3bf0 2024-01-15T11:25:53.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40e2ffd700 1 -- 172.21.15.113:0/1380321208 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e8005430 con 0x7f40f40b3bf0 2024-01-15T11:25:53.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40f898a700 1 -- 172.21.15.113:0/1380321208 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f40ce930 con 0x7f40f40b3bf0 2024-01-15T11:25:53.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40e2ffd700 1 -- 172.21.15.113:0/1380321208 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e80038c0 con 0x7f40f40b3bf0 2024-01-15T11:25:53.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.604+0000 7f40f898a700 1 -- 172.21.15.113:0/1380321208 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f40ceef0 con 0x7f40f40b3bf0 2024-01-15T11:25:53.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.605+0000 7f40f898a700 1 -- 172.21.15.113:0/1380321208 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f4069020 con 0x7f40f40b3bf0 2024-01-15T11:25:53.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.605+0000 7f40e2ffd700 1 -- 172.21.15.113:0/1380321208 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f40e8003a20 con 0x7f40f40b3bf0 2024-01-15T11:25:53.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.606+0000 7f40e2ffd700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f40d406b350 0x7f40d406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.606+0000 7f40f259c700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f40d406b350 0x7f40d406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.606+0000 7f40e2ffd700 1 -- 172.21.15.113:0/1380321208 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40e8088e90 con 0x7f40f40b3bf0 2024-01-15T11:25:53.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.606+0000 7f40f259c700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f40d406b350 0x7f40d406d810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f40e4014040 tx=0x7f40e4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:53.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.611+0000 7f40e2ffd700 1 -- 172.21.15.113:0/1380321208 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f40e8053d90 con 0x7f40f40b3bf0 2024-01-15T11:25:53.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.801+0000 7f40f898a700 1 -- 172.21.15.113:0/1380321208 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f40f40d18f0 con 0x7f40f40b3bf0 2024-01-15T11:25:53.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.801+0000 7f40e2ffd700 1 -- 172.21.15.113:0/1380321208 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f40e8017020 con 0x7f40f40b3bf0 2024-01-15T11:25:53.800 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-15T11:25:53.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.803+0000 7f40f898a700 1 -- 172.21.15.113:0/1380321208 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f40d406b350 msgr2=0x7f40d406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.803+0000 7f40f898a700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f40d406b350 0x7f40d406d810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f40e4014040 tx=0x7f40e4008030 comp rx=0 tx=0).stop 2024-01-15T11:25:53.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.803+0000 7f40f898a700 1 -- 172.21.15.113:0/1380321208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40f40b3bf0 msgr2=0x7f40f415a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.803+0000 7f40f898a700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40f40b3bf0 0x7f40f415a0c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f40e800c510 tx=0x7f40e800c820 comp rx=0 tx=0).stop 2024-01-15T11:25:53.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.803+0000 7f40f898a700 1 -- 172.21.15.113:0/1380321208 shutdown_connections 2024-01-15T11:25:53.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.803+0000 7f40f898a700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f40d406b350 0x7f40d406d810 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.803+0000 7f40f898a700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40f415e0c0 0x7f40f40cf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.803+0000 7f40f898a700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f40f40746a0 0x7f40f414e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.803+0000 7f40f898a700 1 --2- 172.21.15.113:0/1380321208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40f40b3bf0 0x7f40f415a0c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.803+0000 7f40f898a700 1 -- 172.21.15.113:0/1380321208 >> 172.21.15.113:0/1380321208 conn(0x7f40f40d0c00 msgr2=0x7f40f40b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:53.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.803+0000 7f40f898a700 1 -- 172.21.15.113:0/1380321208 shutdown_connections 2024-01-15T11:25:53.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.803+0000 7f40f898a700 1 -- 172.21.15.113:0/1380321208 wait complete. 2024-01-15T11:25:53.814 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T11:25:53.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.940+0000 7f076f3d8700 1 Processor -- start 2024-01-15T11:25:53.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.940+0000 7f076f3d8700 1 -- start start 2024-01-15T11:25:53.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076f3d8700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0768161690 0x7f07680d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076f3d8700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0768161a70 0x7f07680d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076f3d8700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f07680d1730 0x7f076809aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076f3d8700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07680d0a70 con 0x7f0768161690 2024-01-15T11:25:53.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076f3d8700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07680d1c10 con 0x7f07680d1730 2024-01-15T11:25:53.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076f3d8700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07680b2ab0 con 0x7f0768161a70 2024-01-15T11:25:53.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076d174700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0768161690 0x7f07680d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076d174700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0768161690 0x7f07680d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45050/0 (socket says 172.21.15.113:45050) 2024-01-15T11:25:53.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076d174700 1 -- 172.21.15.113:0/3088304639 learned_addr learned my addr 172.21.15.113:0/3088304639 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:53.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076c973700 1 --2- 172.21.15.113:0/3088304639 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0768161a70 0x7f07680d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076d174700 1 -- 172.21.15.113:0/3088304639 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0768161a70 msgr2=0x7f07680d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:53.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076d975700 1 --2- 172.21.15.113:0/3088304639 >> v2:172.21.15.141:3300/0 conn(0x7f07680d1730 0x7f076809aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076d174700 1 --2- 172.21.15.113:0/3088304639 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0768161a70 0x7f07680d2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076d174700 1 -- 172.21.15.113:0/3088304639 >> v2:172.21.15.141:3300/0 conn(0x7f07680d1730 msgr2=0x7f076809aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:53.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076d174700 1 --2- 172.21.15.113:0/3088304639 >> v2:172.21.15.141:3300/0 conn(0x7f07680d1730 0x7f076809aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076d174700 1 -- 172.21.15.113:0/3088304639 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07680b2d90 con 0x7f0768161690 2024-01-15T11:25:53.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f076d174700 1 --2- 172.21.15.113:0/3088304639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0768161690 0x7f07680d0f20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0768072be0 tx=0x7f0758009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0ed178db7cdace server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:53.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f075ffff700 1 -- 172.21.15.113:0/3088304639 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075800b870 con 0x7f0768161690 2024-01-15T11:25:53.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f075ffff700 1 -- 172.21.15.113:0/3088304639 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075800b9d0 con 0x7f0768161690 2024-01-15T11:25:53.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.941+0000 7f075ffff700 1 -- 172.21.15.113:0/3088304639 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075800bd60 con 0x7f0768161690 2024-01-15T11:25:53.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.942+0000 7f076f3d8700 1 -- 172.21.15.113:0/3088304639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0768161690 msgr2=0x7f07680d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.942+0000 7f076f3d8700 1 --2- 172.21.15.113:0/3088304639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0768161690 0x7f07680d0f20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0768072be0 tx=0x7f0758009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.942+0000 7f076f3d8700 1 -- 172.21.15.113:0/3088304639 shutdown_connections 2024-01-15T11:25:53.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.942+0000 7f076f3d8700 1 --2- 172.21.15.113:0/3088304639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0768161690 0x7f07680d0f20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.942+0000 7f076f3d8700 1 --2- 172.21.15.113:0/3088304639 >> v2:172.21.15.141:3300/0 conn(0x7f07680d1730 0x7f076809aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.942+0000 7f076f3d8700 1 --2- 172.21.15.113:0/3088304639 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0768161a70 0x7f07680d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.942+0000 7f076f3d8700 1 -- 172.21.15.113:0/3088304639 >> 172.21.15.113:0/3088304639 conn(0x7f07680d1e50 msgr2=0x7f07680d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:53.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.942+0000 7f076f3d8700 1 -- 172.21.15.113:0/3088304639 shutdown_connections 2024-01-15T11:25:53.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.942+0000 7f076f3d8700 1 -- 172.21.15.113:0/3088304639 wait complete. 2024-01-15T11:25:53.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.943+0000 7f076f3d8700 1 Processor -- start 2024-01-15T11:25:53.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.943+0000 7f076f3d8700 1 -- start start 2024-01-15T11:25:53.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076f3d8700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f07680d1730 0x7f07680c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076f3d8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0768161690 0x7f07680c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076f3d8700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0768161a70 0x7f07680cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076f3d8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07680ca810 con 0x7f0768161690 2024-01-15T11:25:53.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076f3d8700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07680c87c0 con 0x7f07680d1730 2024-01-15T11:25:53.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076f3d8700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07680b0870 con 0x7f0768161a70 2024-01-15T11:25:53.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076c973700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0768161690 0x7f07680c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076c973700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0768161690 0x7f07680c80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:45066/0 (socket says 172.21.15.113:45066) 2024-01-15T11:25:53.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076c973700 1 -- 172.21.15.113:0/3647126514 learned_addr learned my addr 172.21.15.113:0/3647126514 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:53.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076d975700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0768161a70 0x7f07680cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076d174700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f07680d1730 0x7f07680c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076c973700 1 -- 172.21.15.113:0/3647126514 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0768161a70 msgr2=0x7f07680cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076c973700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0768161a70 0x7f07680cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076c973700 1 -- 172.21.15.113:0/3647126514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f07680d1730 msgr2=0x7f07680c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076c973700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f07680d1730 0x7f07680c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.944+0000 7f076c973700 1 -- 172.21.15.113:0/3647126514 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0758009550 con 0x7f0768161690 2024-01-15T11:25:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.945+0000 7f076c973700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0768161690 0x7f07680c80f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f076400c7f0 tx=0x7f076400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.945+0000 7f075dffb700 1 -- 172.21.15.113:0/3647126514 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764011800 con 0x7f0768161690 2024-01-15T11:25:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.945+0000 7f075dffb700 1 -- 172.21.15.113:0/3647126514 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0764011960 con 0x7f0768161690 2024-01-15T11:25:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.945+0000 7f076f3d8700 1 -- 172.21.15.113:0/3647126514 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07680b0b10 con 0x7f0768161690 2024-01-15T11:25:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.945+0000 7f075dffb700 1 -- 172.21.15.113:0/3647126514 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764011d10 con 0x7f0768161690 2024-01-15T11:25:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.945+0000 7f076f3d8700 1 -- 172.21.15.113:0/3647126514 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076814cf70 con 0x7f0768161690 2024-01-15T11:25:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.946+0000 7f076f3d8700 1 -- 172.21.15.113:0/3647126514 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07680ca5e0 con 0x7f0768161690 2024-01-15T11:25:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.946+0000 7f075dffb700 1 -- 172.21.15.113:0/3647126514 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0764018360 con 0x7f0768161690 2024-01-15T11:25:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.946+0000 7f075dffb700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f074c06b350 0x7f074c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.947+0000 7f075dffb700 1 -- 172.21.15.113:0/3647126514 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0764089960 con 0x7f0768161690 2024-01-15T11:25:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.947+0000 7f076d174700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f074c06b350 0x7f074c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.947+0000 7f076d174700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f074c06b350 0x7f074c06d810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0758000c00 tx=0x7f0758008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:53.951+0000 7f075dffb700 1 -- 172.21.15.113:0/3647126514 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f07680ca5e0 con 0x7f0768161690 2024-01-15T11:25:54.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.142+0000 7f076f3d8700 1 -- 172.21.15.113:0/3647126514 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f07680ca5e0 con 0x7f0768161690 2024-01-15T11:25:54.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.143+0000 7f075dffb700 1 -- 172.21.15.113:0/3647126514 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f07680ca5e0 con 0x7f0768161690 2024-01-15T11:25:54.142 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-15T11:25:54.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.145+0000 7f076f3d8700 1 -- 172.21.15.113:0/3647126514 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f074c06b350 msgr2=0x7f074c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:54.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.145+0000 7f076f3d8700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f074c06b350 0x7f074c06d810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0758000c00 tx=0x7f0758008030 comp rx=0 tx=0).stop 2024-01-15T11:25:54.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.145+0000 7f076f3d8700 1 -- 172.21.15.113:0/3647126514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0768161690 msgr2=0x7f07680c80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:54.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.145+0000 7f076f3d8700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0768161690 0x7f07680c80f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f076400c7f0 tx=0x7f076400cbb0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.145+0000 7f076f3d8700 1 -- 172.21.15.113:0/3647126514 shutdown_connections 2024-01-15T11:25:54.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.145+0000 7f076f3d8700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f074c06b350 0x7f074c06d810 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.145+0000 7f076f3d8700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0768161a70 0x7f07680cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.145+0000 7f076f3d8700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f07680d1730 0x7f07680c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.145+0000 7f076f3d8700 1 --2- 172.21.15.113:0/3647126514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0768161690 0x7f07680c80f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.145+0000 7f076f3d8700 1 -- 172.21.15.113:0/3647126514 >> 172.21.15.113:0/3647126514 conn(0x7f07680d1e50 msgr2=0x7f07680cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:54.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.145+0000 7f076f3d8700 1 -- 172.21.15.113:0/3647126514 shutdown_connections 2024-01-15T11:25:54.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.145+0000 7f076f3d8700 1 -- 172.21.15.113:0/3647126514 wait complete. 2024-01-15T11:25:54.156 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T11:25:54.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.280+0000 7f5db354e700 1 Processor -- start 2024-01-15T11:25:54.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.281+0000 7f5db354e700 1 -- start start 2024-01-15T11:25:54.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.281+0000 7f5db354e700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac1473a0 0x7f5dac15e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.281+0000 7f5db354e700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5dac06dc20 0x7f5dac073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.281+0000 7f5db354e700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dac15eb00 con 0x7f5dac147780 2024-01-15T11:25:54.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.281+0000 7f5db354e700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dac15ede0 con 0x7f5dac06dc20 2024-01-15T11:25:54.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5db354e700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac15f0c0 con 0x7f5dac1473a0 2024-01-15T11:25:54.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5db0ae9700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f5dac147780 0x7f5dac06c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48274/0 (socket says 172.21.15.113:48274) 2024-01-15T11:25:54.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5db0ae9700 1 -- 172.21.15.113:0/4269478776 learned_addr learned my addr 172.21.15.113:0/4269478776 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:54.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5db12ea700 1 --2- 172.21.15.113:0/4269478776 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac1473a0 0x7f5dac15e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5db1aeb700 1 --2- 172.21.15.113:0/4269478776 >> v2:172.21.15.141:3300/0 conn(0x7f5dac06dc20 0x7f5dac073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806682513 0 0) 0x7f5dac15eb00 con 0x7f5dac147780 2024-01-15T11:25:54.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d980035b0 con 0x7f5dac147780 2024-01-15T11:25:54.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1730969352 0 0) 0x7f5d980035b0 con 0x7f5dac147780 2024-01-15T11:25:54.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac1473a0 msgr2=0x7f5dac15e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:54.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 --2- 172.21.15.113:0/4269478776 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac1473a0 0x7f5dac15e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 >> v2:172.21.15.141:3300/0 conn(0x7f5dac06dc20 msgr2=0x7f5dac073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:54.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 --2- 172.21.15.113:0/4269478776 >> v2:172.21.15.141:3300/0 conn(0x7f5dac06dc20 0x7f5dac073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac15f3a0 con 0x7f5dac147780 2024-01-15T11:25:54.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f5da8001670 con 0x7f5dac147780 2024-01-15T11:25:54.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5db1aeb700 1 --2- 172.21.15.113:0/4269478776 >> v2:172.21.15.141:3300/0 conn(0x7f5dac06dc20 0x7f5dac073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:54.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 >> v1:172.21.15.113:6789/0 conn(0x7f5dac147780 legacy=0x7f5dac06c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:54.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 --2- 172.21.15.113:0/4269478776 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac1473a0 0x7f5d98003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 --2- 172.21.15.113:0/4269478776 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d98003f80 0x7f5d98004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 --2- 172.21.15.113:0/4269478776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d98004d90 0x7f5d98005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac15ede0 con 0x7f5d98004d90 2024-01-15T11:25:54.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac15f0c0 con 0x7f5d98003f80 2024-01-15T11:25:54.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9800ba10 con 0x7f5dac1473a0 2024-01-15T11:25:54.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5db1aeb700 1 --2- 172.21.15.113:0/4269478776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d98004d90 0x7f5d98005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5db12ea700 1 --2- 172.21.15.113:0/4269478776 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac1473a0 0x7f5d98003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.282+0000 7f5db0ae9700 1 --2- 172.21.15.113:0/4269478776 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d98003f80 0x7f5d98004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.283+0000 7f5db1aeb700 1 -- 172.21.15.113:0/4269478776 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac1473a0 msgr2=0x7f5d98003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:54.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.283+0000 7f5db1aeb700 1 --2- 172.21.15.113:0/4269478776 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac1473a0 0x7f5d98003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.283+0000 7f5db1aeb700 1 -- 172.21.15.113:0/4269478776 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d98003f80 msgr2=0x7f5d98004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:54.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.283+0000 7f5db1aeb700 1 --2- 172.21.15.113:0/4269478776 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d98003f80 0x7f5d98004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.283+0000 7f5db1aeb700 1 -- 172.21.15.113:0/4269478776 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d980035b0 con 0x7f5d98004d90 2024-01-15T11:25:54.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.283+0000 7f5db1aeb700 1 --2- 172.21.15.113:0/4269478776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d98004d90 0x7f5d98005210 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5da400e350 tx=0x7f5d98005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59d74b1b23deb00 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:54.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.283+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da400ed90 con 0x7f5d98004d90 2024-01-15T11:25:54.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.283+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da400f040 con 0x7f5d98004d90 2024-01-15T11:25:54.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.283+0000 7f5d9ffff700 1 -- 172.21.15.113:0/4269478776 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da40146a0 con 0x7f5d98004d90 2024-01-15T11:25:54.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.283+0000 7f5db354e700 1 -- 172.21.15.113:0/4269478776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d98004d90 msgr2=0x7f5d98005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:54.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.283+0000 7f5db354e700 1 --2- 172.21.15.113:0/4269478776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d98004d90 0x7f5d98005210 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5da400e350 tx=0x7f5d98005880 comp rx=0 tx=0).stop 2024-01-15T11:25:54.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.283+0000 7f5db1aeb700 1 -- 172.21.15.113:0/4269478776 reap_dead start 2024-01-15T11:25:54.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.284+0000 7f5db354e700 1 -- 172.21.15.113:0/4269478776 shutdown_connections 2024-01-15T11:25:54.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.284+0000 7f5db354e700 1 -- 172.21.15.113:0/4269478776 >> 172.21.15.113:0/4269478776 conn(0x7f5dac0b3130 msgr2=0x7f5dac0d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:54.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.284+0000 7f5db354e700 1 -- 172.21.15.113:0/4269478776 shutdown_connections 2024-01-15T11:25:54.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.284+0000 7f5db354e700 1 -- 172.21.15.113:0/4269478776 wait complete. 2024-01-15T11:25:54.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.285+0000 7f5db354e700 1 Processor -- start 2024-01-15T11:25:54.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.285+0000 7f5db354e700 1 -- start start 2024-01-15T11:25:54.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db354e700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d98004d90 0x7f5dac135880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db354e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dac0aee80 0x7f5dac0af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db354e700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac073f90 0x7f5dac074410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db354e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac0af840 con 0x7f5dac0aee80 2024-01-15T11:25:54.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db354e700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac074950 con 0x7f5d98004d90 2024-01-15T11:25:54.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db354e700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac1361b0 con 0x7f5dac073f90 2024-01-15T11:25:54.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db0ae9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dac0aee80 0x7f5dac0af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db0ae9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dac0aee80 0x7f5dac0af300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33224/0 (socket says 172.21.15.113:33224) 2024-01-15T11:25:54.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db0ae9700 1 -- 172.21.15.113:0/1240007236 learned_addr learned my addr 172.21.15.113:0/1240007236 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:54.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db12ea700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d98004d90 0x7f5dac135880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db1aeb700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac073f90 0x7f5dac074410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db0ae9700 1 -- 172.21.15.113:0/1240007236 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac073f90 msgr2=0x7f5dac074410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:54.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db0ae9700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac073f90 0x7f5dac074410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db0ae9700 1 -- 172.21.15.113:0/1240007236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d98004d90 msgr2=0x7f5dac135880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:54.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db0ae9700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d98004d90 0x7f5dac135880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db0ae9700 1 -- 172.21.15.113:0/1240007236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac136490 con 0x7f5dac0aee80 2024-01-15T11:25:54.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db12ea700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d98004d90 0x7f5dac135880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:54.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db1aeb700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac073f90 0x7f5dac074410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:54.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db0ae9700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dac0aee80 0x7f5dac0af300 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5da8007310 tx=0x7f5da80076d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:54.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5d9dffb700 1 -- 172.21.15.113:0/1240007236 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da80077f0 con 0x7f5dac0aee80 2024-01-15T11:25:54.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5d9dffb700 1 -- 172.21.15.113:0/1240007236 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da800f8d0 con 0x7f5dac0aee80 2024-01-15T11:25:54.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5db354e700 1 -- 172.21.15.113:0/1240007236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dac136770 con 0x7f5dac0aee80 2024-01-15T11:25:54.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.286+0000 7f5d9dffb700 1 -- 172.21.15.113:0/1240007236 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da800fca0 con 0x7f5dac0aee80 2024-01-15T11:25:54.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.287+0000 7f5db354e700 1 -- 172.21.15.113:0/1240007236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dac133b70 con 0x7f5dac0aee80 2024-01-15T11:25:54.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.287+0000 7f5db354e700 1 -- 172.21.15.113:0/1240007236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dac067f60 con 0x7f5dac0aee80 2024-01-15T11:25:54.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.288+0000 7f5d9dffb700 1 -- 172.21.15.113:0/1240007236 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5da801d460 con 0x7f5dac0aee80 2024-01-15T11:25:54.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.288+0000 7f5d9dffb700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f5d9006b350 0x7f5d9006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.288+0000 7f5d9dffb700 1 -- 172.21.15.113:0/1240007236 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5da8089ca0 con 0x7f5dac0aee80 2024-01-15T11:25:54.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.288+0000 7f5db12ea700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f5d9006b350 0x7f5d9006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.289+0000 7f5db12ea700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f5d9006b350 0x7f5d9006d810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009c60 tx=0x7f5da0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:54.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.293+0000 7f5d9dffb700 1 -- 172.21.15.113:0/1240007236 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5da8054b10 con 0x7f5dac0aee80 2024-01-15T11:25:54.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.498+0000 7f5db354e700 1 -- 172.21.15.113:0/1240007236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5dac06c240 con 0x7f5dac0aee80 2024-01-15T11:25:54.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.499+0000 7f5d9dffb700 1 -- 172.21.15.113:0/1240007236 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5da8058130 con 0x7f5dac0aee80 2024-01-15T11:25:54.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.501+0000 7f5db354e700 1 -- 172.21.15.113:0/1240007236 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f5d9006b350 msgr2=0x7f5d9006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:54.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.501+0000 7f5db354e700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f5d9006b350 0x7f5d9006d810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009c60 tx=0x7f5da0008040 comp rx=0 tx=0).stop 2024-01-15T11:25:54.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.501+0000 7f5db354e700 1 -- 172.21.15.113:0/1240007236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dac0aee80 msgr2=0x7f5dac0af300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:54.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.501+0000 7f5db354e700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dac0aee80 0x7f5dac0af300 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5da8007310 tx=0x7f5da80076d0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.501+0000 7f5db354e700 1 -- 172.21.15.113:0/1240007236 shutdown_connections 2024-01-15T11:25:54.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.501+0000 7f5db354e700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f5d9006b350 0x7f5d9006d810 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.501+0000 7f5db354e700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5dac073f90 0x7f5dac074410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.501+0000 7f5db354e700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d98004d90 0x7f5dac135880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.501+0000 7f5db354e700 1 --2- 172.21.15.113:0/1240007236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dac0aee80 0x7f5dac0af300 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.501+0000 7f5db354e700 1 -- 172.21.15.113:0/1240007236 >> 172.21.15.113:0/1240007236 conn(0x7f5dac0b3130 msgr2=0x7f5dac06e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:54.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.501+0000 7f5db354e700 1 -- 172.21.15.113:0/1240007236 shutdown_connections 2024-01-15T11:25:54.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.501+0000 7f5db354e700 1 -- 172.21.15.113:0/1240007236 wait complete. 2024-01-15T11:25:54.501 INFO:teuthology.orchestra.run.smithi113.stderr:listed 0 entries 2024-01-15T11:25:54.512 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-01-15T11:25:54.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:54.512 DEBUG:teuthology.orchestra.run.smithi113:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-01-15T11:25:54.550 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi113.front.sepia.ceph.com 2024-01-15T11:25:54.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:54.550 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T11:25:54.608 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:25:54.608 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:54.629 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:25:54.629 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:54.690 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:25:54.690 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:25:54.750 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:54.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.873+0000 7f11ef932700 1 Processor -- start 2024-01-15T11:25:54.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.873+0000 7f11ef932700 1 -- start start 2024-01-15T11:25:54.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ef932700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f11e815ad00 0x7f11e806ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ef932700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f11e80b3d20 0x7f11e806d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ef932700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e806f240 con 0x7f11e80b2050 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ef932700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e815b0e0 con 0x7f11e80b3d20 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ef932700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e806dc60 con 0x7f11e815ad00 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ececd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f11e80b3d20 0x7f11e806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11edecf700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f11e80b2050 0x7f11e80cd890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48280/0 (socket says 172.21.15.113:48280) 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11edecf700 1 -- 172.21.15.113:0/2679175675 learned_addr learned my addr 172.21.15.113:0/2679175675 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ececd700 1 -- 172.21.15.113:0/2679175675 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f11e815ad00 msgr2=0x7f11e806ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ed6ce700 1 --2- 172.21.15.113:0/2679175675 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f11e815ad00 0x7f11e806ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ececd700 1 --2- 172.21.15.113:0/2679175675 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f11e815ad00 0x7f11e806ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ececd700 1 -- 172.21.15.113:0/2679175675 >> v1:172.21.15.113:6789/0 conn(0x7f11e80b2050 legacy=0x7f11e80cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11dffff700 1 -- 172.21.15.113:0/2679175675 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882626529 0 0) 0x7f11e806f240 con 0x7f11e80b2050 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ececd700 1 -- 172.21.15.113:0/2679175675 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e806df40 con 0x7f11e80b3d20 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ed6ce700 1 --2- 172.21.15.113:0/2679175675 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f11e815ad00 0x7f11e806ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:54.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11ececd700 1 --2- 172.21.15.113:0/2679175675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11e80b3d20 0x7f11e806d360 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f11d8009840 tx=0x7f11d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9ac565cdc1b675 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:54.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11dffff700 1 -- 172.21.15.113:0/2679175675 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8011830 con 0x7f11e80b3d20 2024-01-15T11:25:54.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11dffff700 1 -- 172.21.15.113:0/2679175675 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d8011990 con 0x7f11e80b3d20 2024-01-15T11:25:54.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.874+0000 7f11dffff700 1 -- 172.21.15.113:0/2679175675 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8011d20 con 0x7f11e80b3d20 2024-01-15T11:25:54.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.875+0000 7f11ef932700 1 -- 172.21.15.113:0/2679175675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11e80b3d20 msgr2=0x7f11e806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:54.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.875+0000 7f11ef932700 1 --2- 172.21.15.113:0/2679175675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11e80b3d20 0x7f11e806d360 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f11d8009840 tx=0x7f11d8009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:54.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.875+0000 7f11ef932700 1 -- 172.21.15.113:0/2679175675 shutdown_connections 2024-01-15T11:25:54.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.875+0000 7f11ef932700 1 --2- 172.21.15.113:0/2679175675 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f11e815ad00 0x7f11e806ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.875+0000 7f11ef932700 1 --2- 172.21.15.113:0/2679175675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11e80b3d20 0x7f11e806d360 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.875+0000 7f11ef932700 1 -- 172.21.15.113:0/2679175675 >> 172.21.15.113:0/2679175675 conn(0x7f11e80743b0 msgr2=0x7f11e806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:54.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.875+0000 7f11ef932700 1 -- 172.21.15.113:0/2679175675 shutdown_connections 2024-01-15T11:25:54.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.875+0000 7f11ef932700 1 -- 172.21.15.113:0/2679175675 wait complete. 2024-01-15T11:25:54.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.876+0000 7f11ef932700 1 Processor -- start 2024-01-15T11:25:54.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.876+0000 7f11ef932700 1 -- start start 2024-01-15T11:25:54.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11ef932700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f11e80b2050 0x7f11e815de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11ef932700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e80b3d20 0x7f11e8151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11ef932700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11e815ad00 0x7f11e80aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11ef932700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e8152250 con 0x7f11e815ad00 2024-01-15T11:25:54.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11ef932700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e81524d0 con 0x7f11e80b3d20 2024-01-15T11:25:54.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11ef932700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e81527b0 con 0x7f11e80b2050 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11edecf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11e815ad00 0x7f11e80aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11edecf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11e815ad00 0x7f11e80aed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33246/0 (socket says 172.21.15.113:33246) 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11edecf700 1 -- 172.21.15.113:0/972579030 learned_addr learned my addr 172.21.15.113:0/972579030 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11ed6ce700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f11e80b2050 0x7f11e815de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11ececd700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e80b3d20 0x7f11e8151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11edecf700 1 -- 172.21.15.113:0/972579030 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f11e80b2050 msgr2=0x7f11e815de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11edecf700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f11e80b2050 0x7f11e815de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11edecf700 1 -- 172.21.15.113:0/972579030 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e80b3d20 msgr2=0x7f11e8151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11edecf700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e80b3d20 0x7f11e8151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11edecf700 1 -- 172.21.15.113:0/972579030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d80094a0 con 0x7f11e815ad00 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.877+0000 7f11ececd700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e80b3d20 0x7f11e8151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.878+0000 7f11edecf700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11e815ad00 0x7f11e80aed10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f11e000ee40 tx=0x7f11e000c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.878+0000 7f11ddffb700 1 -- 172.21.15.113:0/972579030 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e00070d0 con 0x7f11e815ad00 2024-01-15T11:25:54.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.878+0000 7f11ddffb700 1 -- 172.21.15.113:0/972579030 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e0007250 con 0x7f11e815ad00 2024-01-15T11:25:54.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.878+0000 7f11ef932700 1 -- 172.21.15.113:0/972579030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e8152a90 con 0x7f11e815ad00 2024-01-15T11:25:54.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.878+0000 7f11ddffb700 1 -- 172.21.15.113:0/972579030 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0007600 con 0x7f11e815ad00 2024-01-15T11:25:54.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.878+0000 7f11ef932700 1 -- 172.21.15.113:0/972579030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e8153050 con 0x7f11e815ad00 2024-01-15T11:25:54.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.879+0000 7f11ef932700 1 -- 172.21.15.113:0/972579030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e8067f60 con 0x7f11e815ad00 2024-01-15T11:25:54.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.879+0000 7f11ddffb700 1 -- 172.21.15.113:0/972579030 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f11e0007760 con 0x7f11e815ad00 2024-01-15T11:25:54.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.880+0000 7f11ddffb700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f11cc06b350 0x7f11cc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:54.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.880+0000 7f11ed6ce700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f11cc06b350 0x7f11cc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:54.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.880+0000 7f11ddffb700 1 -- 172.21.15.113:0/972579030 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11e008cad0 con 0x7f11e815ad00 2024-01-15T11:25:54.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.880+0000 7f11ed6ce700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f11cc06b350 0x7f11cc06d810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f11e4006fd0 tx=0x7f11e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:54.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:54.884+0000 7f11ddffb700 1 -- 172.21.15.113:0/972579030 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f11e00579d0 con 0x7f11e815ad00 2024-01-15T11:25:55.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.076+0000 7f11ef932700 1 -- 172.21.15.113:0/972579030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11e806b7d0 con 0x7f11e815ad00 2024-01-15T11:25:55.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.077+0000 7f11ddffb700 1 -- 172.21.15.113:0/972579030 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f11e0015020 con 0x7f11e815ad00 2024-01-15T11:25:55.076 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:55.077 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:17.247907+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.079+0000 7f11ef932700 1 -- 172.21.15.113:0/972579030 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f11cc06b350 msgr2=0x7f11cc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.079+0000 7f11ef932700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f11cc06b350 0x7f11cc06d810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f11e4006fd0 tx=0x7f11e4008040 comp rx=0 tx=0).stop 2024-01-15T11:25:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.079+0000 7f11ef932700 1 -- 172.21.15.113:0/972579030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11e815ad00 msgr2=0x7f11e80aed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.079+0000 7f11ef932700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11e815ad00 0x7f11e80aed10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f11e000ee40 tx=0x7f11e000c550 comp rx=0 tx=0).stop 2024-01-15T11:25:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.079+0000 7f11ef932700 1 -- 172.21.15.113:0/972579030 shutdown_connections 2024-01-15T11:25:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.079+0000 7f11ef932700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f11cc06b350 0x7f11cc06d810 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.079+0000 7f11ef932700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11e815ad00 0x7f11e80aed10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.079+0000 7f11ef932700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f11e80b2050 0x7f11e815de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.079+0000 7f11ef932700 1 --2- 172.21.15.113:0/972579030 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11e80b3d20 0x7f11e8151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.079+0000 7f11ef932700 1 -- 172.21.15.113:0/972579030 >> 172.21.15.113:0/972579030 conn(0x7f11e80743b0 msgr2=0x7f11e80afd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:55.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.079+0000 7f11ef932700 1 -- 172.21.15.113:0/972579030 shutdown_connections 2024-01-15T11:25:55.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.079+0000 7f11ef932700 1 -- 172.21.15.113:0/972579030 wait complete. 2024-01-15T11:25:55.091 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T11:25:13.398220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T11:25:14.589439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:55.091 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:55.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.217+0000 7fc1c0d4f700 1 Processor -- start 2024-01-15T11:25:55.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.217+0000 7fc1c0d4f700 1 -- start start 2024-01-15T11:25:55.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1c0d4f700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc1bc161a70 0x7fc1bc0d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1c0d4f700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1bc09b110 0x7fc1bc0d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1c0d4f700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1bc0d0a70 con 0x7fc1bc161690 2024-01-15T11:25:55.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1c0d4f700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1bc0d1c10 con 0x7fc1bc161a70 2024-01-15T11:25:55.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1c0d4f700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0b2ab0 con 0x7fc1bc09b110 2024-01-15T11:25:55.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1ba59c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fc1bc161690 0x7fc1bc09aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48290/0 (socket says 172.21.15.113:48290) 2024-01-15T11:25:55.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1ba59c700 1 -- 172.21.15.113:0/3318859241 learned_addr learned my addr 172.21.15.113:0/3318859241 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:55.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1bad9d700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1bc09b110 0x7fc1bc0d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b9d9b700 1 --2- 172.21.15.113:0/3318859241 >> v2:172.21.15.141:3300/0 conn(0x7fc1bc161a70 0x7fc1bc0d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789347893 0 0) 0x7fc1bc0d0a70 con 0x7fc1bc161690 2024-01-15T11:25:55.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a40035b0 con 0x7fc1bc161690 2024-01-15T11:25:55.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2745303978 0 0) 0x7fc1a40035b0 con 0x7fc1bc161690 2024-01-15T11:25:55.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1bc09b110 msgr2=0x7fc1bc0d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:55.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1bc09b110 0x7fc1bc0d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 >> v2:172.21.15.141:3300/0 conn(0x7fc1bc161a70 msgr2=0x7fc1bc0d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:55.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 --2- 172.21.15.113:0/3318859241 >> v2:172.21.15.141:3300/0 conn(0x7fc1bc161a70 0x7fc1bc0d0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc0b2d90 con 0x7fc1bc161690 2024-01-15T11:25:55.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fc1b0001670 con 0x7fc1bc161690 2024-01-15T11:25:55.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 >> v1:172.21.15.113:6789/0 conn(0x7fc1bc161690 legacy=0x7fc1bc09aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1a4003a00 0x7fc1a4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1bc09b110 0x7fc1a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1a4004d90 0x7fc1a4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0d1c10 con 0x7fc1a4003a00 2024-01-15T11:25:55.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0b2ab0 con 0x7fc1a4004d90 2024-01-15T11:25:55.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a400ba10 con 0x7fc1bc09b110 2024-01-15T11:25:55.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.218+0000 7fc1ba59c700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1a4003a00 0x7fc1a4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1b9d9b700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1bc09b110 0x7fc1a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1bad9d700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1a4004d90 0x7fc1a4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1ba59c700 1 -- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1bc09b110 msgr2=0x7fc1a4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:55.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1ba59c700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1bc09b110 0x7fc1a4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1ba59c700 1 -- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1a4004d90 msgr2=0x7fc1a4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:55.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1ba59c700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1a4004d90 0x7fc1a4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1ba59c700 1 -- 172.21.15.113:0/3318859241 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a40035b0 con 0x7fc1a4003a00 2024-01-15T11:25:55.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1b9d9b700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1bc09b110 0x7fc1a4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:55.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1bad9d700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1a4004d90 0x7fc1a4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:55.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1ba59c700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1a4003a00 0x7fc1a4003e20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000c350 tx=0x7fc1a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7969c27a3718abb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:55.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b000ce50 con 0x7fc1a4003a00 2024-01-15T11:25:55.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b0005960 con 0x7fc1a4003a00 2024-01-15T11:25:55.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1b959a700 1 -- 172.21.15.113:0/3318859241 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b0005d10 con 0x7fc1a4003a00 2024-01-15T11:25:55.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3318859241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1a4003a00 msgr2=0x7fc1a4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1c0d4f700 1 --2- 172.21.15.113:0/3318859241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1a4003a00 0x7fc1a4003e20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000c350 tx=0x7fc1a4005880 comp rx=0 tx=0).stop 2024-01-15T11:25:55.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.219+0000 7fc1bad9d700 1 -- 172.21.15.113:0/3318859241 reap_dead start 2024-01-15T11:25:55.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.220+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3318859241 shutdown_connections 2024-01-15T11:25:55.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.220+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3318859241 >> 172.21.15.113:0/3318859241 conn(0x7fc1bc0d1e50 msgr2=0x7fc1bc0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:55.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.220+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3318859241 shutdown_connections 2024-01-15T11:25:55.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.220+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3318859241 wait complete. 2024-01-15T11:25:55.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.221+0000 7fc1c0d4f700 1 Processor -- start 2024-01-15T11:25:55.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.221+0000 7fc1c0d4f700 1 -- start start 2024-01-15T11:25:55.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1c0d4f700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1a4003a00 0x7fc1bc0b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1c0d4f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1bc0b6960 0x7fc1bc0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1c0d4f700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1bc0b16c0 0x7fc1bc0c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1c0d4f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0b6540 con 0x7fc1bc0b6960 2024-01-15T11:25:55.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1c0d4f700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0b1b70 con 0x7fc1bc0b16c0 2024-01-15T11:25:55.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1c0d4f700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0b5ba0 con 0x7fc1a4003a00 2024-01-15T11:25:55.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1b9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1bc0b6960 0x7fc1bc0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1b9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1bc0b6960 0x7fc1bc0b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33274/0 (socket says 172.21.15.113:33274) 2024-01-15T11:25:55.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1b9d9b700 1 -- 172.21.15.113:0/3922445836 learned_addr learned my addr 172.21.15.113:0/3922445836 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:55.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1bad9d700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1bc0b16c0 0x7fc1bc0c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1ba59c700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1a4003a00 0x7fc1bc0b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1b9d9b700 1 -- 172.21.15.113:0/3922445836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1a4003a00 msgr2=0x7fc1bc0b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1b9d9b700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1a4003a00 0x7fc1bc0b0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1b9d9b700 1 -- 172.21.15.113:0/3922445836 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1bc0b16c0 msgr2=0x7fc1bc0c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1b9d9b700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1bc0b16c0 0x7fc1bc0c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1b9d9b700 1 -- 172.21.15.113:0/3922445836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc0b5e80 con 0x7fc1bc0b6960 2024-01-15T11:25:55.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1bad9d700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1bc0b16c0 0x7fc1bc0c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:55.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1b9d9b700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1bc0b6960 0x7fc1bc0b1180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac010980 tx=0x7fc1ac010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:55.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1aaffd700 1 -- 172.21.15.113:0/3922445836 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac004020 con 0x7fc1bc0b6960 2024-01-15T11:25:55.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1aaffd700 1 -- 172.21.15.113:0/3922445836 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac004180 con 0x7fc1bc0b6960 2024-01-15T11:25:55.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3922445836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1bc0b6160 con 0x7fc1bc0b6960 2024-01-15T11:25:55.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1aaffd700 1 -- 172.21.15.113:0/3922445836 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac004510 con 0x7fc1bc0b6960 2024-01-15T11:25:55.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.222+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3922445836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1bc0b0760 con 0x7fc1bc0b6960 2024-01-15T11:25:55.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.223+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3922445836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1bc0d16b0 con 0x7fc1bc0b6960 2024-01-15T11:25:55.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.224+0000 7fc1aaffd700 1 -- 172.21.15.113:0/3922445836 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc1ac00e8d0 con 0x7fc1bc0b6960 2024-01-15T11:25:55.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.224+0000 7fc1aaffd700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc19c06b350 0x7fc19c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.224+0000 7fc1aaffd700 1 -- 172.21.15.113:0/3922445836 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1ac089e20 con 0x7fc1bc0b6960 2024-01-15T11:25:55.232 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.224+0000 7fc1ba59c700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc19c06b350 0x7fc19c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.232 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.225+0000 7fc1ba59c700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc19c06b350 0x7fc19c06d810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000c270 tx=0x7fc1b000a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:55.232 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.229+0000 7fc1aaffd700 1 -- 172.21.15.113:0/3922445836 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1ac054d20 con 0x7fc1bc0b6960 2024-01-15T11:25:55.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.420+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3922445836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1bc069020 con 0x7fc1bc0b6960 2024-01-15T11:25:55.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.420+0000 7fc1aaffd700 1 -- 172.21.15.113:0/3922445836 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1ac058340 con 0x7fc1bc0b6960 2024-01-15T11:25:55.420 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:55.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.422+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3922445836 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc19c06b350 msgr2=0x7fc19c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.422+0000 7fc1c0d4f700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc19c06b350 0x7fc19c06d810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000c270 tx=0x7fc1b000a000 comp rx=0 tx=0).stop 2024-01-15T11:25:55.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.422+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3922445836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1bc0b6960 msgr2=0x7fc1bc0b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.422+0000 7fc1c0d4f700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1bc0b6960 0x7fc1bc0b1180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac010980 tx=0x7fc1ac010d40 comp rx=0 tx=0).stop 2024-01-15T11:25:55.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.422+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3922445836 shutdown_connections 2024-01-15T11:25:55.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.422+0000 7fc1c0d4f700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc19c06b350 0x7fc19c06d810 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.422+0000 7fc1c0d4f700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1bc0b16c0 0x7fc1bc0c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.422+0000 7fc1c0d4f700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1a4003a00 0x7fc1bc0b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.422+0000 7fc1c0d4f700 1 --2- 172.21.15.113:0/3922445836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1bc0b6960 0x7fc1bc0b1180 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.422+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3922445836 >> 172.21.15.113:0/3922445836 conn(0x7fc1bc0d1e50 msgr2=0x7fc1bc0cc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:55.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.422+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3922445836 shutdown_connections 2024-01-15T11:25:55.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.422+0000 7fc1c0d4f700 1 -- 172.21.15.113:0/3922445836 wait complete. 2024-01-15T11:25:55.434 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T11:25:55.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.559+0000 7fad11c12700 1 Processor -- start 2024-01-15T11:25:55.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.559+0000 7fad11c12700 1 -- start start 2024-01-15T11:25:55.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.559+0000 7fad11c12700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fad0c161690 0x7fad0c0d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.559+0000 7fad11c12700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fad0c09b110 0x7fad0c0d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.559+0000 7fad11c12700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0c0d0a70 con 0x7fad0c161a70 2024-01-15T11:25:55.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.559+0000 7fad11c12700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad0c0d1c10 con 0x7fad0c09b110 2024-01-15T11:25:55.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.559+0000 7fad11c12700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad0c0b2ab0 con 0x7fad0c161690 2024-01-15T11:25:55.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0bfff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fad0c09b110 0x7fad0c0d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0bfff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fad0c09b110 0x7fad0c0d2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33278/0 (socket says 172.21.15.113:33278) 2024-01-15T11:25:55.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0bfff700 1 -- 172.21.15.113:0/3360778651 learned_addr learned my addr 172.21.15.113:0/3360778651 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:55.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0b7fe700 1 --2- 172.21.15.113:0/3360778651 >> v2:172.21.15.141:3300/0 conn(0x7fad0c161690 0x7fad0c0d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0bfff700 1 -- 172.21.15.113:0/3360778651 >> v2:172.21.15.141:3300/0 conn(0x7fad0c161690 msgr2=0x7fad0c0d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:55.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0bfff700 1 --2- 172.21.15.113:0/3360778651 >> v2:172.21.15.141:3300/0 conn(0x7fad0c161690 0x7fad0c0d0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0bfff700 1 -- 172.21.15.113:0/3360778651 >> v1:172.21.15.141:6789/0 conn(0x7fad0c161a70 legacy=0x7fad0c09aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:55.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0bfff700 1 -- 172.21.15.113:0/3360778651 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c0b2d90 con 0x7fad0c09b110 2024-01-15T11:25:55.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0b7fe700 1 --2- 172.21.15.113:0/3360778651 >> v2:172.21.15.141:3300/0 conn(0x7fad0c161690 0x7fad0c0d0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:55.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0bfff700 1 --2- 172.21.15.113:0/3360778651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad0c09b110 0x7fad0c0d2970 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fad0c072010 tx=0x7fad0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d735a6ada9d0a58a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:55.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0a7fc700 1 -- 172.21.15.113:0/3360778651 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0400e070 con 0x7fad0c09b110 2024-01-15T11:25:55.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0a7fc700 1 -- 172.21.15.113:0/3360778651 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad04003a20 con 0x7fad0c09b110 2024-01-15T11:25:55.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.560+0000 7fad0a7fc700 1 -- 172.21.15.113:0/3360778651 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad04003dd0 con 0x7fad0c09b110 2024-01-15T11:25:55.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.561+0000 7fad11c12700 1 -- 172.21.15.113:0/3360778651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad0c09b110 msgr2=0x7fad0c0d2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.561+0000 7fad11c12700 1 --2- 172.21.15.113:0/3360778651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad0c09b110 0x7fad0c0d2970 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fad0c072010 tx=0x7fad0400b330 comp rx=0 tx=0).stop 2024-01-15T11:25:55.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.561+0000 7fad11c12700 1 -- 172.21.15.113:0/3360778651 shutdown_connections 2024-01-15T11:25:55.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.561+0000 7fad11c12700 1 --2- 172.21.15.113:0/3360778651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad0c09b110 0x7fad0c0d2970 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.561+0000 7fad11c12700 1 --2- 172.21.15.113:0/3360778651 >> v2:172.21.15.141:3300/0 conn(0x7fad0c161690 0x7fad0c0d0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.561+0000 7fad11c12700 1 -- 172.21.15.113:0/3360778651 >> 172.21.15.113:0/3360778651 conn(0x7fad0c0d1e50 msgr2=0x7fad0c0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:55.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.561+0000 7fad11c12700 1 -- 172.21.15.113:0/3360778651 shutdown_connections 2024-01-15T11:25:55.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.561+0000 7fad11c12700 1 -- 172.21.15.113:0/3360778651 wait complete. 2024-01-15T11:25:55.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.562+0000 7fad11c12700 1 Processor -- start 2024-01-15T11:25:55.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.562+0000 7fad11c12700 1 -- start start 2024-01-15T11:25:55.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad11c12700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fad0c09b110 0x7fad0c0b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad11c12700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad0c161690 0x7fad0c152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad11c12700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad0c161a70 0x7fad0c153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad11c12700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c153870 con 0x7fad0c161a70 2024-01-15T11:25:55.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad11c12700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c0c7370 con 0x7fad0c161690 2024-01-15T11:25:55.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad11c12700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c0c6760 con 0x7fad0c09b110 2024-01-15T11:25:55.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad0bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad0c161a70 0x7fad0c153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad0bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad0c161a70 0x7fad0c153330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33282/0 (socket says 172.21.15.113:33282) 2024-01-15T11:25:55.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad0bfff700 1 -- 172.21.15.113:0/1541240731 learned_addr learned my addr 172.21.15.113:0/1541240731 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:55.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad0b7fe700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fad0c09b110 0x7fad0c0b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad0affd700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad0c161690 0x7fad0c152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad0bfff700 1 -- 172.21.15.113:0/1541240731 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fad0c09b110 msgr2=0x7fad0c0b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad0bfff700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fad0c09b110 0x7fad0c0b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad0bfff700 1 -- 172.21.15.113:0/1541240731 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad0c161690 msgr2=0x7fad0c152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad0bfff700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad0c161690 0x7fad0c152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad0bfff700 1 -- 172.21.15.113:0/1541240731 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad04009b30 con 0x7fad0c161a70 2024-01-15T11:25:55.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad0b7fe700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fad0c09b110 0x7fad0c0b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:55.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.563+0000 7fad0bfff700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad0c161a70 0x7fad0c153330 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fad04015040 tx=0x7fad04004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:55.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.564+0000 7fad03fff700 1 -- 172.21.15.113:0/1541240731 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad04004440 con 0x7fad0c161a70 2024-01-15T11:25:55.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.564+0000 7fad03fff700 1 -- 172.21.15.113:0/1541240731 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad040045a0 con 0x7fad0c161a70 2024-01-15T11:25:55.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.564+0000 7fad03fff700 1 -- 172.21.15.113:0/1541240731 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad04004950 con 0x7fad0c161a70 2024-01-15T11:25:55.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.564+0000 7fad11c12700 1 -- 172.21.15.113:0/1541240731 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad0c0c6a40 con 0x7fad0c161a70 2024-01-15T11:25:55.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.564+0000 7fad11c12700 1 -- 172.21.15.113:0/1541240731 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad0c0c7000 con 0x7fad0c161a70 2024-01-15T11:25:55.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.565+0000 7fad11c12700 1 -- 172.21.15.113:0/1541240731 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0c0c7520 con 0x7fad0c161a70 2024-01-15T11:25:55.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.565+0000 7fad03fff700 1 -- 172.21.15.113:0/1541240731 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fad04012430 con 0x7fad0c161a70 2024-01-15T11:25:55.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.565+0000 7fad03fff700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7facec06b350 0x7facec06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.566+0000 7fad03fff700 1 -- 172.21.15.113:0/1541240731 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad0408b0d0 con 0x7fad0c161a70 2024-01-15T11:25:55.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.566+0000 7fad0b7fe700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7facec06b350 0x7facec06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.566+0000 7fad0b7fe700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7facec06b350 0x7facec06d810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7facf4008910 tx=0x7facf4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:55.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.570+0000 7fad03fff700 1 -- 172.21.15.113:0/1541240731 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad0c0c7520 con 0x7fad0c161a70 2024-01-15T11:25:55.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.758+0000 7fad11c12700 1 -- 172.21.15.113:0/1541240731 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fad0c0c74e0 con 0x7fad0c161a70 2024-01-15T11:25:55.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.758+0000 7fad03fff700 1 -- 172.21.15.113:0/1541240731 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fad04078020 con 0x7fad0c161a70 2024-01-15T11:25:55.758 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-15T11:25:55.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.761+0000 7fad11c12700 1 -- 172.21.15.113:0/1541240731 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7facec06b350 msgr2=0x7facec06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.761+0000 7fad11c12700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7facec06b350 0x7facec06d810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7facf4008910 tx=0x7facf4008040 comp rx=0 tx=0).stop 2024-01-15T11:25:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.761+0000 7fad11c12700 1 -- 172.21.15.113:0/1541240731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad0c161a70 msgr2=0x7fad0c153330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.761+0000 7fad11c12700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad0c161a70 0x7fad0c153330 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fad04015040 tx=0x7fad04004ad0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.761+0000 7fad11c12700 1 -- 172.21.15.113:0/1541240731 shutdown_connections 2024-01-15T11:25:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.761+0000 7fad11c12700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7facec06b350 0x7facec06d810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.761+0000 7fad11c12700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad0c161a70 0x7fad0c153330 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.761+0000 7fad11c12700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fad0c09b110 0x7fad0c0b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.761+0000 7fad11c12700 1 --2- 172.21.15.113:0/1541240731 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fad0c161690 0x7fad0c152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.761+0000 7fad11c12700 1 -- 172.21.15.113:0/1541240731 >> 172.21.15.113:0/1541240731 conn(0x7fad0c0d1e50 msgr2=0x7fad0c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.761+0000 7fad11c12700 1 -- 172.21.15.113:0/1541240731 shutdown_connections 2024-01-15T11:25:55.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.761+0000 7fad11c12700 1 -- 172.21.15.113:0/1541240731 wait complete. 2024-01-15T11:25:55.772 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T11:25:55.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.898+0000 7fb6fc942700 1 Processor -- start 2024-01-15T11:25:55.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.898+0000 7fb6fc942700 1 -- start start 2024-01-15T11:25:55.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6fc942700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb6f80746a0 0x7fb6f80b0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6fc942700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f80ae300 con 0x7fb6f815c030 2024-01-15T11:25:55.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6fc942700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f815fb00 con 0x7fb6f80d0630 2024-01-15T11:25:55.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6fc942700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f806e350 con 0x7fb6f80746a0 2024-01-15T11:25:55.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f659c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb6f815c030 0x7fb6f806dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48300/0 (socket says 172.21.15.113:48300) 2024-01-15T11:25:55.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f659c700 1 -- 172.21.15.113:0/1064257070 learned_addr learned my addr 172.21.15.113:0/1064257070 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:55.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f6d9d700 1 --2- 172.21.15.113:0/1064257070 >> v2:172.21.15.141:3300/0 conn(0x7fb6f80746a0 0x7fb6f80b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684158405 0 0) 0x7fb6f80ae300 con 0x7fb6f815c030 2024-01-15T11:25:55.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6e00035b0 con 0x7fb6f815c030 2024-01-15T11:25:55.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 393943029 0 0) 0x7fb6e00035b0 con 0x7fb6f815c030 2024-01-15T11:25:55.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 >> v2:172.21.15.141:3300/0 conn(0x7fb6f80746a0 msgr2=0x7fb6f80b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:55.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 --2- 172.21.15.113:0/1064257070 >> v2:172.21.15.141:3300/0 conn(0x7fb6f80746a0 0x7fb6f80b0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 >> v1:172.21.15.141:6789/0 conn(0x7fb6f80d0630 legacy=0x7fb6f8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f815c410 con 0x7fb6f815c030 2024-01-15T11:25:55.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fb6e8001670 con 0x7fb6f815c030 2024-01-15T11:25:55.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 >> v1:172.21.15.113:6789/0 conn(0x7fb6f815c030 legacy=0x7fb6f806dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 --2- 172.21.15.113:0/1064257070 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e0003a00 0x7fb6e0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 --2- 172.21.15.113:0/1064257070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e0004850 0x7fb6e0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 --2- 172.21.15.113:0/1064257070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6e0005210 0x7fb6e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.899+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f806e350 con 0x7fb6e0005210 2024-01-15T11:25:55.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e000c1b0 con 0x7fb6e0004850 2024-01-15T11:25:55.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e000c370 con 0x7fb6e0003a00 2024-01-15T11:25:55.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f6d9d700 1 --2- 172.21.15.113:0/1064257070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6e0005210 0x7fb6e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f5d9b700 1 --2- 172.21.15.113:0/1064257070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e0004850 0x7fb6e0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f659c700 1 --2- 172.21.15.113:0/1064257070 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e0003a00 0x7fb6e0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f6d9d700 1 -- 172.21.15.113:0/1064257070 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e0003a00 msgr2=0x7fb6e0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:55.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f6d9d700 1 --2- 172.21.15.113:0/1064257070 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e0003a00 0x7fb6e0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f6d9d700 1 -- 172.21.15.113:0/1064257070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e0004850 msgr2=0x7fb6e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:55.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f6d9d700 1 --2- 172.21.15.113:0/1064257070 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6e0004850 0x7fb6e0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f6d9d700 1 -- 172.21.15.113:0/1064257070 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e00035b0 con 0x7fb6e0005210 2024-01-15T11:25:55.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f659c700 1 --2- 172.21.15.113:0/1064257070 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e0003a00 0x7fb6e0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:55.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f6d9d700 1 -- 172.21.15.113:0/1064257070 reap_dead start 2024-01-15T11:25:55.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f6d9d700 1 --2- 172.21.15.113:0/1064257070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6e0005210 0x7fb6e000ba10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8071890 tx=0x7fb6e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85741b557d8efba6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:55.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0003c20 con 0x7fb6e0005210 2024-01-15T11:25:55.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f000f040 con 0x7fb6e0005210 2024-01-15T11:25:55.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.900+0000 7fb6f559a700 1 -- 172.21.15.113:0/1064257070 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f000e9d0 con 0x7fb6e0005210 2024-01-15T11:25:55.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.901+0000 7fb6fc942700 1 -- 172.21.15.113:0/1064257070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6e0005210 msgr2=0x7fb6e000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.901+0000 7fb6fc942700 1 --2- 172.21.15.113:0/1064257070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6e0005210 0x7fb6e000ba10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8071890 tx=0x7fb6e0005880 comp rx=0 tx=0).stop 2024-01-15T11:25:55.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.901+0000 7fb6fc942700 1 -- 172.21.15.113:0/1064257070 shutdown_connections 2024-01-15T11:25:55.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.901+0000 7fb6fc942700 1 --2- 172.21.15.113:0/1064257070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6e0005210 0x7fb6e000ba10 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.901+0000 7fb6fc942700 1 -- 172.21.15.113:0/1064257070 >> 172.21.15.113:0/1064257070 conn(0x7fb6f80d0c00 msgr2=0x7fb6f80d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:55.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.901+0000 7fb6fc942700 1 -- 172.21.15.113:0/1064257070 shutdown_connections 2024-01-15T11:25:55.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.901+0000 7fb6fc942700 1 -- 172.21.15.113:0/1064257070 wait complete. 2024-01-15T11:25:55.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.902+0000 7fb6fc942700 1 Processor -- start 2024-01-15T11:25:55.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.902+0000 7fb6fc942700 1 -- start start 2024-01-15T11:25:55.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6fc942700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e0005210 0x7fb6f814dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6fc942700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6f814e190 0x7fb6f80b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6fc942700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6f80b4230 0x7fb6f80b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6fc942700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f8154900 con 0x7fb6f80b4230 2024-01-15T11:25:55.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6fc942700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f814ea20 con 0x7fb6f814e190 2024-01-15T11:25:55.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6fc942700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f814ed00 con 0x7fb6e0005210 2024-01-15T11:25:55.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6f6d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6f80b4230 0x7fb6f80b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6f6d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6f80b4230 0x7fb6f80b46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33302/0 (socket says 172.21.15.113:33302) 2024-01-15T11:25:55.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6f6d9d700 1 -- 172.21.15.113:0/1593561940 learned_addr learned my addr 172.21.15.113:0/1593561940 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:55.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6f5d9b700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6f814e190 0x7fb6f80b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6f659c700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e0005210 0x7fb6f814dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6f6d9d700 1 -- 172.21.15.113:0/1593561940 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e0005210 msgr2=0x7fb6f814dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6f6d9d700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e0005210 0x7fb6f814dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6f6d9d700 1 -- 172.21.15.113:0/1593561940 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6f814e190 msgr2=0x7fb6f80b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:55.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6f6d9d700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6f814e190 0x7fb6f80b3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:55.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6f6d9d700 1 -- 172.21.15.113:0/1593561940 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f814efe0 con 0x7fb6f80b4230 2024-01-15T11:25:55.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6f5d9b700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6f814e190 0x7fb6f80b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:55.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6f6d9d700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6f80b4230 0x7fb6f80b46b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb6f80d0ba0 tx=0x7fb6f0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:55.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.903+0000 7fb6e6ffd700 1 -- 172.21.15.113:0/1593561940 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0004430 con 0x7fb6f80b4230 2024-01-15T11:25:55.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.904+0000 7fb6e6ffd700 1 -- 172.21.15.113:0/1593561940 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f0003e90 con 0x7fb6f80b4230 2024-01-15T11:25:55.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.904+0000 7fb6fc942700 1 -- 172.21.15.113:0/1593561940 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f814f2c0 con 0x7fb6f80b4230 2024-01-15T11:25:55.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.904+0000 7fb6e6ffd700 1 -- 172.21.15.113:0/1593561940 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f00047f0 con 0x7fb6f80b4230 2024-01-15T11:25:55.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.904+0000 7fb6fc942700 1 -- 172.21.15.113:0/1593561940 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f8151080 con 0x7fb6f80b4230 2024-01-15T11:25:55.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.904+0000 7fb6fc942700 1 -- 172.21.15.113:0/1593561940 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f814f780 con 0x7fb6f80b4230 2024-01-15T11:25:55.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.905+0000 7fb6e6ffd700 1 -- 172.21.15.113:0/1593561940 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb6f0014480 con 0x7fb6f80b4230 2024-01-15T11:25:55.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.905+0000 7fb6e6ffd700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb6d806b380 0x7fb6d806d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:55.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.905+0000 7fb6f659c700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb6d806b380 0x7fb6d806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:55.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.905+0000 7fb6e6ffd700 1 -- 172.21.15.113:0/1593561940 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6f008b230 con 0x7fb6f80b4230 2024-01-15T11:25:55.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.906+0000 7fb6f659c700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb6d806b380 0x7fb6d806d840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8005a20 tx=0x7fb6e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:55.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:55.910+0000 7fb6e6ffd700 1 -- 172.21.15.113:0/1593561940 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6f0056250 con 0x7fb6f80b4230 2024-01-15T11:25:56.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.099+0000 7fb6fc942700 1 -- 172.21.15.113:0/1593561940 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb6f8069020 con 0x7fb6f80b4230 2024-01-15T11:25:56.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.100+0000 7fb6e6ffd700 1 -- 172.21.15.113:0/1593561940 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb6f0059870 con 0x7fb6f80b4230 2024-01-15T11:25:56.099 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-15T11:25:56.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.102+0000 7fb6fc942700 1 -- 172.21.15.113:0/1593561940 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb6d806b380 msgr2=0x7fb6d806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.102+0000 7fb6fc942700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb6d806b380 0x7fb6d806d840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8005a20 tx=0x7fb6e8009040 comp rx=0 tx=0).stop 2024-01-15T11:25:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.102+0000 7fb6fc942700 1 -- 172.21.15.113:0/1593561940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6f80b4230 msgr2=0x7fb6f80b46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.102+0000 7fb6fc942700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6f80b4230 0x7fb6f80b46b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb6f80d0ba0 tx=0x7fb6f0009750 comp rx=0 tx=0).stop 2024-01-15T11:25:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.102+0000 7fb6fc942700 1 -- 172.21.15.113:0/1593561940 shutdown_connections 2024-01-15T11:25:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.102+0000 7fb6fc942700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb6d806b380 0x7fb6d806d840 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.102+0000 7fb6fc942700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6f80b4230 0x7fb6f80b46b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.102+0000 7fb6fc942700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb6e0005210 0x7fb6f814dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.102+0000 7fb6fc942700 1 --2- 172.21.15.113:0/1593561940 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb6f814e190 0x7fb6f80b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.102+0000 7fb6fc942700 1 -- 172.21.15.113:0/1593561940 >> 172.21.15.113:0/1593561940 conn(0x7fb6f80d0c00 msgr2=0x7fb6f806dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.102+0000 7fb6fc942700 1 -- 172.21.15.113:0/1593561940 shutdown_connections 2024-01-15T11:25:56.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.102+0000 7fb6fc942700 1 -- 172.21.15.113:0/1593561940 wait complete. 2024-01-15T11:25:56.114 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T11:25:56.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.238+0000 7efd1868b700 1 Processor -- start 2024-01-15T11:25:56.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.238+0000 7efd1868b700 1 -- start start 2024-01-15T11:25:56.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.239+0000 7efd1868b700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7efd100b50c0 0x7efd100cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.239+0000 7efd1868b700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1006bf50 con 0x7efd1006f460 2024-01-15T11:25:56.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.239+0000 7efd1868b700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1015b030 con 0x7efd1015ac50 2024-01-15T11:25:56.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.239+0000 7efd1868b700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd1006e810 con 0x7efd100b50c0 2024-01-15T11:25:56.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15c26700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7efd1006f460 0x7efd100b49b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48304/0 (socket says 172.21.15.113:48304) 2024-01-15T11:25:56.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15c26700 1 -- 172.21.15.113:0/2452031411 learned_addr learned my addr 172.21.15.113:0/2452031411 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:56.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd16c28700 1 --2- 172.21.15.113:0/2452031411 >> v2:172.21.15.141:3300/0 conn(0x7efd100b50c0 0x7efd100cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456236506 0 0) 0x7efd1006bf50 con 0x7efd1006f460 2024-01-15T11:25:56.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcfc0035b0 con 0x7efd1006f460 2024-01-15T11:25:56.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3948521103 0 0) 0x7efcfc0035b0 con 0x7efd1006f460 2024-01-15T11:25:56.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 >> v2:172.21.15.141:3300/0 conn(0x7efd100b50c0 msgr2=0x7efd100cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:56.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 --2- 172.21.15.113:0/2452031411 >> v2:172.21.15.141:3300/0 conn(0x7efd100b50c0 0x7efd100cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 >> v1:172.21.15.141:6789/0 conn(0x7efd1015ac50 legacy=0x7efd1006c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1006b340 con 0x7efd1006f460 2024-01-15T11:25:56.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7efd0c001670 con 0x7efd1006f460 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 >> v1:172.21.15.113:6789/0 conn(0x7efd1006f460 legacy=0x7efd100b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 --2- 172.21.15.113:0/2452031411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcfc003a00 0x7efcfc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 --2- 172.21.15.113:0/2452031411 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efcfc004850 0x7efcfc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 --2- 172.21.15.113:0/2452031411 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcfc005210 0x7efcfc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1006e810 con 0x7efcfc003a00 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc00c0b0 con 0x7efcfc004850 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc00c270 con 0x7efcfc005210 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd16427700 1 --2- 172.21.15.113:0/2452031411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcfc003a00 0x7efcfc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd15c26700 1 --2- 172.21.15.113:0/2452031411 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efcfc004850 0x7efcfc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.240+0000 7efd16c28700 1 --2- 172.21.15.113:0/2452031411 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcfc005210 0x7efcfc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd16427700 1 -- 172.21.15.113:0/2452031411 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcfc005210 msgr2=0x7efcfc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd16427700 1 --2- 172.21.15.113:0/2452031411 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcfc005210 0x7efcfc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd16427700 1 -- 172.21.15.113:0/2452031411 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efcfc004850 msgr2=0x7efcfc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:56.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd16427700 1 --2- 172.21.15.113:0/2452031411 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efcfc004850 0x7efcfc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd16427700 1 -- 172.21.15.113:0/2452031411 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc0035b0 con 0x7efcfc003a00 2024-01-15T11:25:56.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd16c28700 1 -- 172.21.15.113:0/2452031411 reap_dead start 2024-01-15T11:25:56.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd16427700 1 --2- 172.21.15.113:0/2452031411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcfc003a00 0x7efcfc003e20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efd000017c0 tx=0x7efcfc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43164e614e8f9673 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:56.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd000052d0 con 0x7efcfc003a00 2024-01-15T11:25:56.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd00005430 con 0x7efcfc003a00 2024-01-15T11:25:56.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd15425700 1 -- 172.21.15.113:0/2452031411 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd000038c0 con 0x7efcfc003a00 2024-01-15T11:25:56.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd1868b700 1 -- 172.21.15.113:0/2452031411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcfc003a00 msgr2=0x7efcfc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd1868b700 1 --2- 172.21.15.113:0/2452031411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcfc003a00 0x7efcfc003e20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efd000017c0 tx=0x7efcfc005880 comp rx=0 tx=0).stop 2024-01-15T11:25:56.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd1868b700 1 -- 172.21.15.113:0/2452031411 shutdown_connections 2024-01-15T11:25:56.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd1868b700 1 --2- 172.21.15.113:0/2452031411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcfc003a00 0x7efcfc003e20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd1868b700 1 -- 172.21.15.113:0/2452031411 >> 172.21.15.113:0/2452031411 conn(0x7efd100b3130 msgr2=0x7efd100d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:56.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.241+0000 7efd1868b700 1 -- 172.21.15.113:0/2452031411 shutdown_connections 2024-01-15T11:25:56.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.242+0000 7efd1868b700 1 -- 172.21.15.113:0/2452031411 wait complete. 2024-01-15T11:25:56.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.243+0000 7efd1868b700 1 Processor -- start 2024-01-15T11:25:56.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.243+0000 7efd1868b700 1 -- start start 2024-01-15T11:25:56.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.243+0000 7efd1868b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcfc003a00 0x7efd1014eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.243+0000 7efd1868b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd10155590 0x7efd1015c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.243+0000 7efd1868b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd10155a10 0x7efd10150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.243+0000 7efd1868b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd100afe60 con 0x7efd10155a10 2024-01-15T11:25:56.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.243+0000 7efd1868b700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd100af1e0 con 0x7efd10155590 2024-01-15T11:25:56.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.243+0000 7efd1868b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd100af4c0 con 0x7efcfc003a00 2024-01-15T11:25:56.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd16c28700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd10155a10 0x7efd10150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd16c28700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd10155a10 0x7efd10150410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33322/0 (socket says 172.21.15.113:33322) 2024-01-15T11:25:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd16c28700 1 -- 172.21.15.113:0/3881286810 learned_addr learned my addr 172.21.15.113:0/3881286810 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd16427700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcfc003a00 0x7efd1014eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd15c26700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd10155590 0x7efd1015c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd16c28700 1 -- 172.21.15.113:0/3881286810 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcfc003a00 msgr2=0x7efd1014eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd16c28700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcfc003a00 0x7efd1014eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd16c28700 1 -- 172.21.15.113:0/3881286810 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd10155590 msgr2=0x7efd1015c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd16c28700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd10155590 0x7efd1015c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd16c28700 1 -- 172.21.15.113:0/3881286810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd100af7a0 con 0x7efd10155a10 2024-01-15T11:25:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd16c28700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd10155a10 0x7efd10150410 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efd1006dc40 tx=0x7efd08007f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd06ffd700 1 -- 172.21.15.113:0/3881286810 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0800e070 con 0x7efd10155a10 2024-01-15T11:25:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd06ffd700 1 -- 172.21.15.113:0/3881286810 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd08003cf0 con 0x7efd10155a10 2024-01-15T11:25:56.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd1868b700 1 -- 172.21.15.113:0/3881286810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd100afa80 con 0x7efd10155a10 2024-01-15T11:25:56.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd06ffd700 1 -- 172.21.15.113:0/3881286810 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08003ea0 con 0x7efd10155a10 2024-01-15T11:25:56.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.244+0000 7efd1868b700 1 -- 172.21.15.113:0/3881286810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1015cc80 con 0x7efd10155a10 2024-01-15T11:25:56.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.245+0000 7efd1868b700 1 -- 172.21.15.113:0/3881286810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd1015b210 con 0x7efd10155a10 2024-01-15T11:25:56.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.246+0000 7efd06ffd700 1 -- 172.21.15.113:0/3881286810 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efd080116d0 con 0x7efd10155a10 2024-01-15T11:25:56.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.246+0000 7efd06ffd700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efcf406b350 0x7efcf406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.246+0000 7efd06ffd700 1 -- 172.21.15.113:0/3881286810 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd08089e30 con 0x7efd10155a10 2024-01-15T11:25:56.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.246+0000 7efd16427700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efcf406b350 0x7efcf406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.246+0000 7efd16427700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efcf406b350 0x7efcf406d810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd00000c00 tx=0x7efd00009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:56.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.252+0000 7efd06ffd700 1 -- 172.21.15.113:0/3881286810 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd1015b210 con 0x7efd10155a10 2024-01-15T11:25:56.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.465+0000 7efd1868b700 1 -- 172.21.15.113:0/3881286810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efd1015b1c0 con 0x7efd10155a10 2024-01-15T11:25:56.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.466+0000 7efd06ffd700 1 -- 172.21.15.113:0/3881286810 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7efd08054d30 con 0x7efd10155a10 2024-01-15T11:25:56.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.468+0000 7efd1868b700 1 -- 172.21.15.113:0/3881286810 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efcf406b350 msgr2=0x7efcf406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.468+0000 7efd1868b700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efcf406b350 0x7efcf406d810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd00000c00 tx=0x7efd00009030 comp rx=0 tx=0).stop 2024-01-15T11:25:56.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.468+0000 7efd1868b700 1 -- 172.21.15.113:0/3881286810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd10155a10 msgr2=0x7efd10150410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.468+0000 7efd1868b700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd10155a10 0x7efd10150410 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efd1006dc40 tx=0x7efd08007f20 comp rx=0 tx=0).stop 2024-01-15T11:25:56.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.468+0000 7efd1868b700 1 -- 172.21.15.113:0/3881286810 shutdown_connections 2024-01-15T11:25:56.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.468+0000 7efd1868b700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efcf406b350 0x7efcf406d810 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.468+0000 7efd1868b700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd10155a10 0x7efd10150410 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.468+0000 7efd1868b700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efcfc003a00 0x7efd1014eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.468+0000 7efd1868b700 1 --2- 172.21.15.113:0/3881286810 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efd10155590 0x7efd1015c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.468+0000 7efd1868b700 1 -- 172.21.15.113:0/3881286810 >> 172.21.15.113:0/3881286810 conn(0x7efd100b3130 msgr2=0x7efd10074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:56.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.468+0000 7efd1868b700 1 -- 172.21.15.113:0/3881286810 shutdown_connections 2024-01-15T11:25:56.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.468+0000 7efd1868b700 1 -- 172.21.15.113:0/3881286810 wait complete. 2024-01-15T11:25:56.469 INFO:teuthology.orchestra.run.smithi113.stderr:listed 0 entries 2024-01-15T11:25:56.479 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-01-15T11:25:56.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:56.480 DEBUG:teuthology.orchestra.run.smithi141:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-01-15T11:25:56.526 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi141.front.sepia.ceph.com 2024-01-15T11:25:56.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:25:56.526 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T11:25:56.546 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T11:25:56.547 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T11:25:56.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.669+0000 7f16152b4700 1 Processor -- start 2024-01-15T11:25:56.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.669+0000 7f16152b4700 1 -- start start 2024-01-15T11:25:56.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f16152b4700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f161015e0f0 0x7f16100b9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f16152b4700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f16100b2b60 0x7f16100ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f16152b4700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f161006bc00 con 0x7f16100cf0d0 2024-01-15T11:25:56.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f16152b4700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16100cf560 con 0x7f16100b2b60 2024-01-15T11:25:56.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f16152b4700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f161015e4d0 con 0x7f161015e0f0 2024-01-15T11:25:56.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160e7fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f16100b2b60 0x7f16100ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160e7fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f16100b2b60 0x7f16100ceb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33332/0 (socket says 172.21.15.113:33332) 2024-01-15T11:25:56.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160e7fc700 1 -- 172.21.15.113:0/3066457354 learned_addr learned my addr 172.21.15.113:0/3066457354 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:56.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160effd700 1 --2- 172.21.15.113:0/3066457354 >> v2:172.21.15.141:3300/0 conn(0x7f161015e0f0 0x7f16100b9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160e7fc700 1 -- 172.21.15.113:0/3066457354 >> v2:172.21.15.141:3300/0 conn(0x7f161015e0f0 msgr2=0x7f16100b9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:56.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160e7fc700 1 --2- 172.21.15.113:0/3066457354 >> v2:172.21.15.141:3300/0 conn(0x7f161015e0f0 0x7f16100b9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160e7fc700 1 -- 172.21.15.113:0/3066457354 >> v1:172.21.15.141:6789/0 conn(0x7f16100cf0d0 legacy=0x7f1610161bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:56.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160e7fc700 1 -- 172.21.15.113:0/3066457354 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161006aff0 con 0x7f16100b2b60 2024-01-15T11:25:56.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160effd700 1 --2- 172.21.15.113:0/3066457354 >> v2:172.21.15.141:3300/0 conn(0x7f161015e0f0 0x7f16100b9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:56.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160e7fc700 1 --2- 172.21.15.113:0/3066457354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100b2b60 0x7f16100ceb90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f161008c270 tx=0x7f1604009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12efe40e3b0c302 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:56.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160dffb700 1 -- 172.21.15.113:0/3066457354 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1604004230 con 0x7f16100b2b60 2024-01-15T11:25:56.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160dffb700 1 -- 172.21.15.113:0/3066457354 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1604004390 con 0x7f16100b2b60 2024-01-15T11:25:56.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.670+0000 7f160dffb700 1 -- 172.21.15.113:0/3066457354 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16040036a0 con 0x7f16100b2b60 2024-01-15T11:25:56.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.671+0000 7f16152b4700 1 -- 172.21.15.113:0/3066457354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100b2b60 msgr2=0x7f16100ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.671+0000 7f16152b4700 1 --2- 172.21.15.113:0/3066457354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100b2b60 0x7f16100ceb90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f161008c270 tx=0x7f1604009bd0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.671+0000 7f16152b4700 1 -- 172.21.15.113:0/3066457354 shutdown_connections 2024-01-15T11:25:56.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.671+0000 7f16152b4700 1 --2- 172.21.15.113:0/3066457354 >> v2:172.21.15.141:3300/0 conn(0x7f161015e0f0 0x7f16100b9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.671+0000 7f16152b4700 1 --2- 172.21.15.113:0/3066457354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100b2b60 0x7f16100ceb90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.671+0000 7f16152b4700 1 -- 172.21.15.113:0/3066457354 >> 172.21.15.113:0/3066457354 conn(0x7f16100b5ce0 msgr2=0x7f16100b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:56.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.671+0000 7f16152b4700 1 -- 172.21.15.113:0/3066457354 shutdown_connections 2024-01-15T11:25:56.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.671+0000 7f16152b4700 1 -- 172.21.15.113:0/3066457354 wait complete. 2024-01-15T11:25:56.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.672+0000 7f16152b4700 1 Processor -- start 2024-01-15T11:25:56.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.672+0000 7f16152b4700 1 -- start start 2024-01-15T11:25:56.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.673+0000 7f16152b4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100b2b60 0x7f161015b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.673+0000 7f16152b4700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16100cf0d0 0x7f16100cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.673+0000 7f16152b4700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f161015e0f0 0x7f16100c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.673+0000 7f16152b4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161015b810 con 0x7f16100b2b60 2024-01-15T11:25:56.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.673+0000 7f16152b4700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16100cd9d0 con 0x7f16100cf0d0 2024-01-15T11:25:56.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.673+0000 7f16152b4700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16100cdcb0 con 0x7f161015e0f0 2024-01-15T11:25:56.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.673+0000 7f160effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100b2b60 0x7f161015b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.673+0000 7f160effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100b2b60 0x7f161015b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33342/0 (socket says 172.21.15.113:33342) 2024-01-15T11:25:56.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.673+0000 7f160effd700 1 -- 172.21.15.113:0/1454408660 learned_addr learned my addr 172.21.15.113:0/1454408660 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:56.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.673+0000 7f160f7fe700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f161015e0f0 0x7f16100c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.673+0000 7f160e7fc700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16100cf0d0 0x7f16100cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.674+0000 7f160effd700 1 -- 172.21.15.113:0/1454408660 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f161015e0f0 msgr2=0x7f16100c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.674+0000 7f160effd700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f161015e0f0 0x7f16100c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.674+0000 7f160effd700 1 -- 172.21.15.113:0/1454408660 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16100cf0d0 msgr2=0x7f16100cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.674+0000 7f160effd700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16100cf0d0 0x7f16100cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.674+0000 7f160effd700 1 -- 172.21.15.113:0/1454408660 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1604009550 con 0x7f16100b2b60 2024-01-15T11:25:56.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.674+0000 7f160effd700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100b2b60 0x7f161015b2d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f160000c7f0 tx=0x7f160000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:56.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.674+0000 7f15f77fe700 1 -- 172.21.15.113:0/1454408660 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1600011800 con 0x7f16100b2b60 2024-01-15T11:25:56.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.674+0000 7f15f77fe700 1 -- 172.21.15.113:0/1454408660 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1600011960 con 0x7f16100b2b60 2024-01-15T11:25:56.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.674+0000 7f16152b4700 1 -- 172.21.15.113:0/1454408660 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16100c8a00 con 0x7f16100b2b60 2024-01-15T11:25:56.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.674+0000 7f15f77fe700 1 -- 172.21.15.113:0/1454408660 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1600011d10 con 0x7f16100b2b60 2024-01-15T11:25:56.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.674+0000 7f16152b4700 1 -- 172.21.15.113:0/1454408660 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16100c8fc0 con 0x7f16100b2b60 2024-01-15T11:25:56.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.675+0000 7f16152b4700 1 -- 172.21.15.113:0/1454408660 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16100c91e0 con 0x7f16100b2b60 2024-01-15T11:25:56.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.675+0000 7f15f77fe700 1 -- 172.21.15.113:0/1454408660 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1600018360 con 0x7f16100b2b60 2024-01-15T11:25:56.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.676+0000 7f15f77fe700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f15fc06b350 0x7f15fc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:56.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.676+0000 7f15f77fe700 1 -- 172.21.15.113:0/1454408660 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1600089ba0 con 0x7f16100b2b60 2024-01-15T11:25:56.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.676+0000 7f160e7fc700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f15fc06b350 0x7f15fc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:56.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.676+0000 7f160e7fc700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f15fc06b350 0x7f15fc06d810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1604009130 tx=0x7f1604008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:56.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.680+0000 7f15f77fe700 1 -- 172.21.15.113:0/1454408660 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1600054a20 con 0x7f16100b2b60 2024-01-15T11:25:56.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.912+0000 7f16152b4700 1 -- 172.21.15.113:0/1454408660 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16100751e0 con 0x7f16100b2b60 2024-01-15T11:25:56.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.912+0000 7f15f77fe700 1 -- 172.21.15.113:0/1454408660 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f160001f090 con 0x7f16100b2b60 2024-01-15T11:25:56.912 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:56.912 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3164513953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":3164513953},{"type":"v1","addr":"172.21.15.113:6835","nonce":3164513953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/891522731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":891522731},{"type":"v1","addr":"172.21.15.113:6837","nonce":891522731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2158560535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2158560535},{"type":"v1","addr":"172.21.15.141:6834","nonce":2158560535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T11:25:15.611582+0000","modified":"2024-01-15T11:25:18.718623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4300":{"gid":4300,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1012738257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1012738257},{"type":"v1","addr":"172.21.15.141:6835","nonce":1012738257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T11:25:56.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.914+0000 7f16152b4700 1 -- 172.21.15.113:0/1454408660 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f15fc06b350 msgr2=0x7f15fc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.914+0000 7f16152b4700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f15fc06b350 0x7f15fc06d810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1604009130 tx=0x7f1604008030 comp rx=0 tx=0).stop 2024-01-15T11:25:56.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.914+0000 7f16152b4700 1 -- 172.21.15.113:0/1454408660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100b2b60 msgr2=0x7f161015b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:56.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.914+0000 7f16152b4700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100b2b60 0x7f161015b2d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f160000c7f0 tx=0x7f160000cbb0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.915+0000 7f16152b4700 1 -- 172.21.15.113:0/1454408660 shutdown_connections 2024-01-15T11:25:56.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.915+0000 7f16152b4700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f15fc06b350 0x7f15fc06d810 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.915+0000 7f16152b4700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f161015e0f0 0x7f16100c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.915+0000 7f16152b4700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100b2b60 0x7f161015b2d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.915+0000 7f16152b4700 1 --2- 172.21.15.113:0/1454408660 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f16100cf0d0 0x7f16100cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:56.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.915+0000 7f16152b4700 1 -- 172.21.15.113:0/1454408660 >> 172.21.15.113:0/1454408660 conn(0x7f16100b5ce0 msgr2=0x7f16100ce660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:56.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.915+0000 7f16152b4700 1 -- 172.21.15.113:0/1454408660 shutdown_connections 2024-01-15T11:25:56.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:56.915+0000 7f16152b4700 1 -- 172.21.15.113:0/1454408660 wait complete. 2024-01-15T11:25:56.915 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-15T11:25:56.926 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.048+0000 7f309efca700 1 Processor -- start 2024-01-15T11:25:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.048+0000 7f309efca700 1 -- start start 2024-01-15T11:25:57.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f309efca700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f309806f460 0x7f30980b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f309efca700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309806bf50 con 0x7f30980b4ef0 2024-01-15T11:25:57.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f309efca700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309815b030 con 0x7f309815ac50 2024-01-15T11:25:57.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f309efca700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30980b5360 con 0x7f309806f460 2024-01-15T11:25:57.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f309d567700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f30980b4ef0 0x7f30980cdf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48308/0 (socket says 172.21.15.113:48308) 2024-01-15T11:25:57.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f309d567700 1 -- 172.21.15.113:0/1993368744 learned_addr learned my addr 172.21.15.113:0/1993368744 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:57.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803051574 0 0) 0x7f309806bf50 con 0x7f30980b4ef0 2024-01-15T11:25:57.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30880035b0 con 0x7f30980b4ef0 2024-01-15T11:25:57.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f3097fff700 1 --2- 172.21.15.113:0/1993368744 >> v2:172.21.15.141:3300/0 conn(0x7f309806f460 0x7f30980b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866223884 0 0) 0x7f30880035b0 con 0x7f30980b4ef0 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 >> v2:172.21.15.141:3300/0 conn(0x7f309806f460 msgr2=0x7f30980b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 --2- 172.21.15.113:0/1993368744 >> v2:172.21.15.141:3300/0 conn(0x7f309806f460 0x7f30980b49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 >> v1:172.21.15.141:6789/0 conn(0x7f309815ac50 legacy=0x7f309806c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309806e810 con 0x7f30980b4ef0 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f30900019f0 con 0x7f30980b4ef0 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f3097fff700 1 --2- 172.21.15.113:0/1993368744 >> v2:172.21.15.141:3300/0 conn(0x7f309806f460 0x7f30980b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 >> v1:172.21.15.113:6789/0 conn(0x7f30980b4ef0 legacy=0x7f30980cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 --2- 172.21.15.113:0/1993368744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3088003a00 0x7f3088003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 --2- 172.21.15.113:0/1993368744 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3088004850 0x7f3088004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 --2- 172.21.15.113:0/1993368744 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3088005210 0x7f308800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30980b5360 con 0x7f3088003a00 2024-01-15T11:25:57.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308800c160 con 0x7f3088004850 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.049+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308800c320 con 0x7f3088005210 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f309cd66700 1 --2- 172.21.15.113:0/1993368744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3088003a00 0x7f3088003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f3097fff700 1 --2- 172.21.15.113:0/1993368744 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3088004850 0x7f3088004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f309d567700 1 --2- 172.21.15.113:0/1993368744 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3088005210 0x7f308800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f309cd66700 1 -- 172.21.15.113:0/1993368744 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3088005210 msgr2=0x7f308800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f309cd66700 1 --2- 172.21.15.113:0/1993368744 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3088005210 0x7f308800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f309cd66700 1 -- 172.21.15.113:0/1993368744 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3088004850 msgr2=0x7f3088004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f309cd66700 1 --2- 172.21.15.113:0/1993368744 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3088004850 0x7f3088004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f309cd66700 1 -- 172.21.15.113:0/1993368744 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30880035b0 con 0x7f3088003a00 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f309d567700 1 -- 172.21.15.113:0/1993368744 reap_dead start 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f3097fff700 1 --2- 172.21.15.113:0/1993368744 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3088004850 0x7f3088004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f309cd66700 1 --2- 172.21.15.113:0/1993368744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3088003a00 0x7f3088003e20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f308c0017c0 tx=0x7f3088005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca38bb777c57b6f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:57.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c0052d0 con 0x7f3088003a00 2024-01-15T11:25:57.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308c005430 con 0x7f3088003a00 2024-01-15T11:25:57.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.050+0000 7f30977fe700 1 -- 172.21.15.113:0/1993368744 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c0038c0 con 0x7f3088003a00 2024-01-15T11:25:57.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.051+0000 7f309efca700 1 -- 172.21.15.113:0/1993368744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3088003a00 msgr2=0x7f3088003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.051+0000 7f309efca700 1 --2- 172.21.15.113:0/1993368744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3088003a00 0x7f3088003e20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f308c0017c0 tx=0x7f3088005880 comp rx=0 tx=0).stop 2024-01-15T11:25:57.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.051+0000 7f309efca700 1 -- 172.21.15.113:0/1993368744 shutdown_connections 2024-01-15T11:25:57.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.051+0000 7f309efca700 1 --2- 172.21.15.113:0/1993368744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3088003a00 0x7f3088003e20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.051+0000 7f309efca700 1 -- 172.21.15.113:0/1993368744 >> 172.21.15.113:0/1993368744 conn(0x7f30980b3130 msgr2=0x7f30980d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:57.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.051+0000 7f309efca700 1 -- 172.21.15.113:0/1993368744 shutdown_connections 2024-01-15T11:25:57.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.051+0000 7f309efca700 1 -- 172.21.15.113:0/1993368744 wait complete. 2024-01-15T11:25:57.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.052+0000 7f309efca700 1 Processor -- start 2024-01-15T11:25:57.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.052+0000 7f309efca700 1 -- start start 2024-01-15T11:25:57.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f309efca700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3088003a00 0x7f30981f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f309efca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f309815e6a0 0x7f309815eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f309efca700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30981e54e0 0x7f30981e5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f309efca700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309815f060 con 0x7f309815e6a0 2024-01-15T11:25:57.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f309efca700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3098157b60 con 0x7f30981e54e0 2024-01-15T11:25:57.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f309efca700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3098156b00 con 0x7f3088003a00 2024-01-15T11:25:57.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f3097fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f309815e6a0 0x7f309815eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f3097fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f309815e6a0 0x7f309815eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33358/0 (socket says 172.21.15.113:33358) 2024-01-15T11:25:57.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f3097fff700 1 -- 172.21.15.113:0/2480933987 learned_addr learned my addr 172.21.15.113:0/2480933987 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:57.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f3097fff700 1 -- 172.21.15.113:0/2480933987 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3088003a00 msgr2=0x7f30981f1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f3097fff700 1 --2- 172.21.15.113:0/2480933987 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3088003a00 0x7f30981f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f3097fff700 1 -- 172.21.15.113:0/2480933987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30981e54e0 msgr2=0x7f30981e5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f3097fff700 1 --2- 172.21.15.113:0/2480933987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30981e54e0 0x7f30981e5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f3097fff700 1 -- 172.21.15.113:0/2480933987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3098156da0 con 0x7f309815e6a0 2024-01-15T11:25:57.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f3097fff700 1 --2- 172.21.15.113:0/2480933987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f309815e6a0 0x7f309815eb20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3084010870 tx=0x7f3084010c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:57.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f30957fa700 1 -- 172.21.15.113:0/2480933987 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308400e7e0 con 0x7f309815e6a0 2024-01-15T11:25:57.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f30957fa700 1 -- 172.21.15.113:0/2480933987 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308400c050 con 0x7f309815e6a0 2024-01-15T11:25:57.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f30957fa700 1 -- 172.21.15.113:0/2480933987 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308400eb40 con 0x7f309815e6a0 2024-01-15T11:25:57.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f309efca700 1 -- 172.21.15.113:0/2480933987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3098157080 con 0x7f309815e6a0 2024-01-15T11:25:57.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.053+0000 7f309efca700 1 -- 172.21.15.113:0/2480933987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3098157640 con 0x7f309815e6a0 2024-01-15T11:25:57.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.054+0000 7f309efca700 1 -- 172.21.15.113:0/2480933987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30981579b0 con 0x7f309815e6a0 2024-01-15T11:25:57.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.055+0000 7f30957fa700 1 -- 172.21.15.113:0/2480933987 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f308400eca0 con 0x7f309815e6a0 2024-01-15T11:25:57.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.055+0000 7f30957fa700 1 --2- 172.21.15.113:0/2480933987 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f307c06b350 0x7f307c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.055+0000 7f309cd66700 1 --2- 172.21.15.113:0/2480933987 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f307c06b350 0x7f307c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.055+0000 7f30957fa700 1 -- 172.21.15.113:0/2480933987 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f308408a250 con 0x7f309815e6a0 2024-01-15T11:25:57.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.056+0000 7f309cd66700 1 --2- 172.21.15.113:0/2480933987 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f307c06b350 0x7f307c06d810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f308c000c00 tx=0x7f308c011000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:57.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.060+0000 7f30957fa700 1 -- 172.21.15.113:0/2480933987 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f30840550f0 con 0x7f309815e6a0 2024-01-15T11:25:57.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.253+0000 7f309efca700 1 -- 172.21.15.113:0/2480933987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3098067f60 con 0x7f309815e6a0 2024-01-15T11:25:57.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.254+0000 7f30957fa700 1 -- 172.21.15.113:0/2480933987 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16757 (secure 0 0 0) 0x7f3084009480 con 0x7f309815e6a0 2024-01-15T11:25:57.254 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:57.254 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:56.246653+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":1}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:57.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.257+0000 7f309efca700 1 -- 172.21.15.113:0/2480933987 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f307c06b350 msgr2=0x7f307c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.257+0000 7f309efca700 1 --2- 172.21.15.113:0/2480933987 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f307c06b350 0x7f307c06d810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f308c000c00 tx=0x7f308c011000 comp rx=0 tx=0).stop 2024-01-15T11:25:57.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.257+0000 7f309efca700 1 -- 172.21.15.113:0/2480933987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f309815e6a0 msgr2=0x7f309815eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.257+0000 7f309efca700 1 --2- 172.21.15.113:0/2480933987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f309815e6a0 0x7f309815eb20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3084010870 tx=0x7f3084010c30 comp rx=0 tx=0).stop 2024-01-15T11:25:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.258+0000 7f309efca700 1 -- 172.21.15.113:0/2480933987 shutdown_connections 2024-01-15T11:25:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.258+0000 7f309efca700 1 --2- 172.21.15.113:0/2480933987 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f307c06b350 0x7f307c06d810 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.258+0000 7f309efca700 1 --2- 172.21.15.113:0/2480933987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f30981e54e0 0x7f30981e5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.258+0000 7f309efca700 1 --2- 172.21.15.113:0/2480933987 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3088003a00 0x7f30981f1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.258+0000 7f309efca700 1 --2- 172.21.15.113:0/2480933987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f309815e6a0 0x7f309815eb20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.258+0000 7f309efca700 1 -- 172.21.15.113:0/2480933987 >> 172.21.15.113:0/2480933987 conn(0x7f30980b3130 msgr2=0x7f30980b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.258+0000 7f309efca700 1 -- 172.21.15.113:0/2480933987 shutdown_connections 2024-01-15T11:25:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.258+0000 7f309efca700 1 -- 172.21.15.113:0/2480933987 wait complete. 2024-01-15T11:25:57.271 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T11:25:13.398220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T11:25:14.589439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:25:57.272 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:25:57.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.398+0000 7f2bb437a700 1 Processor -- start 2024-01-15T11:25:57.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.398+0000 7f2bb437a700 1 -- start start 2024-01-15T11:25:57.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb437a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2bac0746a0 0x7f2bac0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb437a700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bac0afa60 0x7f2bac0afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb437a700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac145b70 con 0x7f2bac145790 2024-01-15T11:25:57.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb437a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bac0b3bf0 con 0x7f2bac0746a0 2024-01-15T11:25:57.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb437a700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac0b3e40 con 0x7f2bac0afa60 2024-01-15T11:25:57.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb1915700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2bac0746a0 0x7f2bac0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb1915700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2bac0746a0 0x7f2bac0b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33372/0 (socket says 172.21.15.113:33372) 2024-01-15T11:25:57.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb1915700 1 -- 172.21.15.113:0/3006407733 learned_addr learned my addr 172.21.15.113:0/3006407733 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:57.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb1915700 1 -- 172.21.15.113:0/3006407733 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bac0afa60 msgr2=0x7f2bac0afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:57.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb1915700 1 --2- 172.21.15.113:0/3006407733 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bac0afa60 0x7f2bac0afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb1915700 1 -- 172.21.15.113:0/3006407733 >> v1:172.21.15.141:6789/0 conn(0x7f2bac145790 legacy=0x7f2bac073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:57.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb1915700 1 -- 172.21.15.113:0/3006407733 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bac0b4120 con 0x7f2bac0746a0 2024-01-15T11:25:57.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb1915700 1 --2- 172.21.15.113:0/3006407733 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bac0746a0 0x7f2bac0b4c00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80098f0 tx=0x7f2ba8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b75af4f28bcc67d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:57.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb1114700 1 -- 172.21.15.113:0/3006407733 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8004230 con 0x7f2bac0746a0 2024-01-15T11:25:57.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb1114700 1 -- 172.21.15.113:0/3006407733 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba8004390 con 0x7f2bac0746a0 2024-01-15T11:25:57.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.399+0000 7f2bb1114700 1 -- 172.21.15.113:0/3006407733 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80036a0 con 0x7f2bac0746a0 2024-01-15T11:25:57.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.400+0000 7f2bb437a700 1 -- 172.21.15.113:0/3006407733 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bac0746a0 msgr2=0x7f2bac0b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.400+0000 7f2bb437a700 1 --2- 172.21.15.113:0/3006407733 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bac0746a0 0x7f2bac0b4c00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80098f0 tx=0x7f2ba8009c00 comp rx=0 tx=0).stop 2024-01-15T11:25:57.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.400+0000 7f2bb437a700 1 -- 172.21.15.113:0/3006407733 shutdown_connections 2024-01-15T11:25:57.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.400+0000 7f2bb437a700 1 --2- 172.21.15.113:0/3006407733 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bac0afa60 0x7f2bac0afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.400+0000 7f2bb437a700 1 --2- 172.21.15.113:0/3006407733 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bac0746a0 0x7f2bac0b4c00 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.400+0000 7f2bb437a700 1 -- 172.21.15.113:0/3006407733 >> 172.21.15.113:0/3006407733 conn(0x7f2bac0b3250 msgr2=0x7f2bac0d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:57.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.400+0000 7f2bb437a700 1 -- 172.21.15.113:0/3006407733 shutdown_connections 2024-01-15T11:25:57.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.400+0000 7f2bb437a700 1 -- 172.21.15.113:0/3006407733 wait complete. 2024-01-15T11:25:57.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.401+0000 7f2bb437a700 1 Processor -- start 2024-01-15T11:25:57.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.401+0000 7f2bb437a700 1 -- start start 2024-01-15T11:25:57.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb437a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bac0746a0 0x7f2bac140030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb437a700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bac0afa60 0x7f2bac133f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb437a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bac145790 0x7f2bac0c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb437a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac09d240 con 0x7f2bac0746a0 2024-01-15T11:25:57.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb437a700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac0c60a0 con 0x7f2bac145790 2024-01-15T11:25:57.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb437a700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac09c630 con 0x7f2bac0afa60 2024-01-15T11:25:57.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb2116700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bac0746a0 0x7f2bac140030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb2116700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bac0746a0 0x7f2bac140030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33388/0 (socket says 172.21.15.113:33388) 2024-01-15T11:25:57.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb2116700 1 -- 172.21.15.113:0/926522987 learned_addr learned my addr 172.21.15.113:0/926522987 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:57.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb2116700 1 -- 172.21.15.113:0/926522987 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bac0afa60 msgr2=0x7f2bac133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb2116700 1 --2- 172.21.15.113:0/926522987 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bac0afa60 0x7f2bac133f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb2116700 1 -- 172.21.15.113:0/926522987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bac145790 msgr2=0x7f2bac0c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb2116700 1 --2- 172.21.15.113:0/926522987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bac145790 0x7f2bac0c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb2116700 1 -- 172.21.15.113:0/926522987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba8009550 con 0x7f2bac0746a0 2024-01-15T11:25:57.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.402+0000 7f2bb2116700 1 --2- 172.21.15.113:0/926522987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bac0746a0 0x7f2bac140030 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000b470 tx=0x7f2ba000b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:57.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.403+0000 7f2b9effd700 1 -- 172.21.15.113:0/926522987 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0011830 con 0x7f2bac0746a0 2024-01-15T11:25:57.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.403+0000 7f2b9effd700 1 -- 172.21.15.113:0/926522987 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba00119b0 con 0x7f2bac0746a0 2024-01-15T11:25:57.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.403+0000 7f2bb437a700 1 -- 172.21.15.113:0/926522987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bac09c910 con 0x7f2bac0746a0 2024-01-15T11:25:57.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.403+0000 7f2b9effd700 1 -- 172.21.15.113:0/926522987 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0011d60 con 0x7f2bac0746a0 2024-01-15T11:25:57.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.403+0000 7f2bb437a700 1 -- 172.21.15.113:0/926522987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bac09ced0 con 0x7f2bac0746a0 2024-01-15T11:25:57.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.404+0000 7f2bb437a700 1 -- 172.21.15.113:0/926522987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bac0b3990 con 0x7f2bac0746a0 2024-01-15T11:25:57.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.404+0000 7f2b9effd700 1 -- 172.21.15.113:0/926522987 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2ba001a410 con 0x7f2bac0746a0 2024-01-15T11:25:57.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.404+0000 7f2b9effd700 1 --2- 172.21.15.113:0/926522987 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2b9006b380 0x7f2b9006d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.404+0000 7f2b9effd700 1 -- 172.21.15.113:0/926522987 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2ba00897b0 con 0x7f2bac0746a0 2024-01-15T11:25:57.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.404+0000 7f2bb1915700 1 --2- 172.21.15.113:0/926522987 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2b9006b380 0x7f2b9006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.405+0000 7f2bb1915700 1 --2- 172.21.15.113:0/926522987 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2b9006b380 0x7f2b9006d840 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2bac1370c0 tx=0x7f2ba8010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:57.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.409+0000 7f2b9effd700 1 -- 172.21.15.113:0/926522987 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ba0054500 con 0x7f2bac0746a0 2024-01-15T11:25:57.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.597+0000 7f2bb437a700 1 -- 172.21.15.113:0/926522987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2bac069020 con 0x7f2bac0746a0 2024-01-15T11:25:57.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.598+0000 7f2b9effd700 1 -- 172.21.15.113:0/926522987 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ba0057b20 con 0x7f2bac0746a0 2024-01-15T11:25:57.597 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:25:57.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.600+0000 7f2bb437a700 1 -- 172.21.15.113:0/926522987 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2b9006b380 msgr2=0x7f2b9006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.600+0000 7f2bb437a700 1 --2- 172.21.15.113:0/926522987 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2b9006b380 0x7f2b9006d840 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2bac1370c0 tx=0x7f2ba8010000 comp rx=0 tx=0).stop 2024-01-15T11:25:57.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.600+0000 7f2bb437a700 1 -- 172.21.15.113:0/926522987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bac0746a0 msgr2=0x7f2bac140030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.600+0000 7f2bb437a700 1 --2- 172.21.15.113:0/926522987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bac0746a0 0x7f2bac140030 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000b470 tx=0x7f2ba000b830 comp rx=0 tx=0).stop 2024-01-15T11:25:57.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.600+0000 7f2bb437a700 1 -- 172.21.15.113:0/926522987 shutdown_connections 2024-01-15T11:25:57.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.600+0000 7f2bb437a700 1 --2- 172.21.15.113:0/926522987 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2b9006b380 0x7f2b9006d840 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.600+0000 7f2bb437a700 1 --2- 172.21.15.113:0/926522987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2bac145790 0x7f2bac0c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.600+0000 7f2bb437a700 1 --2- 172.21.15.113:0/926522987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bac0746a0 0x7f2bac140030 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.600+0000 7f2bb437a700 1 --2- 172.21.15.113:0/926522987 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2bac0afa60 0x7f2bac133f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.600+0000 7f2bb437a700 1 -- 172.21.15.113:0/926522987 >> 172.21.15.113:0/926522987 conn(0x7f2bac0b3250 msgr2=0x7f2bac073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:57.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.600+0000 7f2bb437a700 1 -- 172.21.15.113:0/926522987 shutdown_connections 2024-01-15T11:25:57.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.600+0000 7f2bb437a700 1 -- 172.21.15.113:0/926522987 wait complete. 2024-01-15T11:25:57.611 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T11:25:57.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.737+0000 7feefa388700 1 Processor -- start 2024-01-15T11:25:57.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.737+0000 7feefa388700 1 -- start start 2024-01-15T11:25:57.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.737+0000 7feefa388700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7feef4161690 0x7feef40d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.737+0000 7feefa388700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef4161a70 0x7feef40d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.737+0000 7feefa388700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef40d0a70 con 0x7feef40d1730 2024-01-15T11:25:57.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.737+0000 7feefa388700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef40d1c10 con 0x7feef4161690 2024-01-15T11:25:57.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.737+0000 7feefa388700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef40b2ab0 con 0x7feef4161a70 2024-01-15T11:25:57.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef3fff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7feef4161690 0x7feef40d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef8925700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7feef40d1730 0x7feef409aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48310/0 (socket says 172.21.15.113:48310) 2024-01-15T11:25:57.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef8925700 1 -- 172.21.15.113:0/289036894 learned_addr learned my addr 172.21.15.113:0/289036894 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:57.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef37fe700 1 --2- 172.21.15.113:0/289036894 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef4161a70 0x7feef40d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef3fff700 1 -- 172.21.15.113:0/289036894 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef4161a70 msgr2=0x7feef40d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:57.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef3fff700 1 --2- 172.21.15.113:0/289036894 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef4161a70 0x7feef40d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef3fff700 1 -- 172.21.15.113:0/289036894 >> v1:172.21.15.113:6789/0 conn(0x7feef40d1730 legacy=0x7feef409aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef3fff700 1 -- 172.21.15.113:0/289036894 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef40b2d90 con 0x7feef4161690 2024-01-15T11:25:57.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef2ffd700 1 -- 172.21.15.113:0/289036894 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474925638 0 0) 0x7feef40d0a70 con 0x7feef40d1730 2024-01-15T11:25:57.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef3fff700 1 --2- 172.21.15.113:0/289036894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef4161690 0x7feef40d0f20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7feee4009840 tx=0x7feee4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4d0d9d9c3e285b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:57.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef2ffd700 1 -- 172.21.15.113:0/289036894 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee4011830 con 0x7feef4161690 2024-01-15T11:25:57.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef2ffd700 1 -- 172.21.15.113:0/289036894 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee4011990 con 0x7feef4161690 2024-01-15T11:25:57.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.738+0000 7feef2ffd700 1 -- 172.21.15.113:0/289036894 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee4011d20 con 0x7feef4161690 2024-01-15T11:25:57.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.739+0000 7feefa388700 1 -- 172.21.15.113:0/289036894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef4161690 msgr2=0x7feef40d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.739+0000 7feefa388700 1 --2- 172.21.15.113:0/289036894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef4161690 0x7feef40d0f20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7feee4009840 tx=0x7feee4009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:57.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.739+0000 7feefa388700 1 -- 172.21.15.113:0/289036894 shutdown_connections 2024-01-15T11:25:57.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.739+0000 7feefa388700 1 --2- 172.21.15.113:0/289036894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef4161690 0x7feef40d0f20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.739+0000 7feefa388700 1 --2- 172.21.15.113:0/289036894 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef4161a70 0x7feef40d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.739+0000 7feefa388700 1 -- 172.21.15.113:0/289036894 >> 172.21.15.113:0/289036894 conn(0x7feef40d1e50 msgr2=0x7feef40d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:57.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.739+0000 7feefa388700 1 -- 172.21.15.113:0/289036894 shutdown_connections 2024-01-15T11:25:57.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.739+0000 7feefa388700 1 -- 172.21.15.113:0/289036894 wait complete. 2024-01-15T11:25:57.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.740+0000 7feefa388700 1 Processor -- start 2024-01-15T11:25:57.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.740+0000 7feefa388700 1 -- start start 2024-01-15T11:25:57.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feefa388700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef40d1730 0x7feef40afbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feefa388700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feef4161690 0x7feef41f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feefa388700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef4161a70 0x7feef40b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feefa388700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef41f4c20 con 0x7feef4161a70 2024-01-15T11:25:57.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feefa388700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef41f81a0 con 0x7feef4161690 2024-01-15T11:25:57.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feefa388700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef40aedf0 con 0x7feef40d1730 2024-01-15T11:25:57.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feef8925700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef4161a70 0x7feef40b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feef8925700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef4161a70 0x7feef40b0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33408/0 (socket says 172.21.15.113:33408) 2024-01-15T11:25:57.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feef8925700 1 -- 172.21.15.113:0/1945429111 learned_addr learned my addr 172.21.15.113:0/1945429111 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:57.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feef37fe700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feef4161690 0x7feef41f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feef3fff700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef40d1730 0x7feef40afbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feef8925700 1 -- 172.21.15.113:0/1945429111 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef40d1730 msgr2=0x7feef40afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feef8925700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef40d1730 0x7feef40afbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feef8925700 1 -- 172.21.15.113:0/1945429111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feef4161690 msgr2=0x7feef41f7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feef8925700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feef4161690 0x7feef41f7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feef8925700 1 -- 172.21.15.113:0/1945429111 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee40094a0 con 0x7feef4161a70 2024-01-15T11:25:57.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feef37fe700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feef4161690 0x7feef41f7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:57.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.741+0000 7feef8925700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef4161a70 0x7feef40b0270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7feeec00cb60 tx=0x7feeec00ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:57.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.742+0000 7feef0ff9700 1 -- 172.21.15.113:0/1945429111 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec00aa60 con 0x7feef4161a70 2024-01-15T11:25:57.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.742+0000 7feef0ff9700 1 -- 172.21.15.113:0/1945429111 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec00abc0 con 0x7feef4161a70 2024-01-15T11:25:57.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.742+0000 7feefa388700 1 -- 172.21.15.113:0/1945429111 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef40af0d0 con 0x7feef4161a70 2024-01-15T11:25:57.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.742+0000 7feef0ff9700 1 -- 172.21.15.113:0/1945429111 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec0070d0 con 0x7feef4161a70 2024-01-15T11:25:57.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.742+0000 7feefa388700 1 -- 172.21.15.113:0/1945429111 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef40af690 con 0x7feef4161a70 2024-01-15T11:25:57.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.742+0000 7feefa388700 1 -- 172.21.15.113:0/1945429111 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef40af8b0 con 0x7feef4161a70 2024-01-15T11:25:57.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.743+0000 7feef0ff9700 1 -- 172.21.15.113:0/1945429111 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feeec007230 con 0x7feef4161a70 2024-01-15T11:25:57.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.743+0000 7feef0ff9700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7feee006b350 0x7feee006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:57.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.744+0000 7feef0ff9700 1 -- 172.21.15.113:0/1945429111 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7feeec089970 con 0x7feef4161a70 2024-01-15T11:25:57.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.744+0000 7feef3fff700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7feee006b350 0x7feee006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:57.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.744+0000 7feef3fff700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7feee006b350 0x7feee006d810 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7feee4000c00 tx=0x7feee401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:57.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.748+0000 7feef0ff9700 1 -- 172.21.15.113:0/1945429111 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feeec054760 con 0x7feef4161a70 2024-01-15T11:25:57.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.937+0000 7feefa388700 1 -- 172.21.15.113:0/1945429111 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feef4068670 con 0x7feef4161a70 2024-01-15T11:25:57.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.937+0000 7feef0ff9700 1 -- 172.21.15.113:0/1945429111 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7feeec0180a0 con 0x7feef4161a70 2024-01-15T11:25:57.936 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-15T11:25:57.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.939+0000 7feefa388700 1 -- 172.21.15.113:0/1945429111 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7feee006b350 msgr2=0x7feee006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.939+0000 7feefa388700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7feee006b350 0x7feee006d810 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7feee4000c00 tx=0x7feee401c040 comp rx=0 tx=0).stop 2024-01-15T11:25:57.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.939+0000 7feefa388700 1 -- 172.21.15.113:0/1945429111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef4161a70 msgr2=0x7feef40b0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:57.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.939+0000 7feefa388700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef4161a70 0x7feef40b0270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7feeec00cb60 tx=0x7feeec00ce70 comp rx=0 tx=0).stop 2024-01-15T11:25:57.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.939+0000 7feefa388700 1 -- 172.21.15.113:0/1945429111 shutdown_connections 2024-01-15T11:25:57.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.939+0000 7feefa388700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7feee006b350 0x7feee006d810 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.939+0000 7feefa388700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef4161a70 0x7feef40b0270 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.939+0000 7feefa388700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7feef40d1730 0x7feef40afbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.939+0000 7feefa388700 1 --2- 172.21.15.113:0/1945429111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feef4161690 0x7feef41f7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:57.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.939+0000 7feefa388700 1 -- 172.21.15.113:0/1945429111 >> 172.21.15.113:0/1945429111 conn(0x7feef40d1e50 msgr2=0x7feef40cce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:57.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.939+0000 7feefa388700 1 -- 172.21.15.113:0/1945429111 shutdown_connections 2024-01-15T11:25:57.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:57.939+0000 7feefa388700 1 -- 172.21.15.113:0/1945429111 wait complete. 2024-01-15T11:25:57.950 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T11:25:58.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.074+0000 7faee1d6d700 1 Processor -- start 2024-01-15T11:25:58.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.074+0000 7faee1d6d700 1 -- start start 2024-01-15T11:25:58.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.075+0000 7faee1d6d700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc0cffc0 0x7faedc0b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.075+0000 7faee1d6d700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7faedc0d03a0 0x7faedc15f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.075+0000 7faee1d6d700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faedc0b3860 con 0x7faedc15f700 2024-01-15T11:25:58.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.075+0000 7faee1d6d700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faedc15fb90 con 0x7faedc0d03a0 2024-01-15T11:25:58.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.075+0000 7faee1d6d700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0b2520 con 0x7faedc0cffc0 2024-01-15T11:25:58.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.075+0000 7faedbfff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7faedc15f700 0x7faedc09aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48326/0 (socket says 172.21.15.113:48326) 2024-01-15T11:25:58.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.075+0000 7faedbfff700 1 -- 172.21.15.113:0/1954434082 learned_addr learned my addr 172.21.15.113:0/1954434082 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.075+0000 7faedaffd700 1 --2- 172.21.15.113:0/1954434082 >> v2:172.21.15.141:3300/0 conn(0x7faedc0d03a0 0x7faedc15f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.075+0000 7faedb7fe700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc0cffc0 0x7faedc0b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.075+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105856982 0 0) 0x7faed4001490 con 0x7faedc15f700 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.075+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec40035b0 con 0x7faedc15f700 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 821556319 0 0) 0x7faec40035b0 con 0x7faedc15f700 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc0cffc0 msgr2=0x7faedc0b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc0cffc0 0x7faedc0b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 >> v2:172.21.15.141:3300/0 conn(0x7faedc0d03a0 msgr2=0x7faedc15f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 --2- 172.21.15.113:0/1954434082 >> v2:172.21.15.141:3300/0 conn(0x7faedc0d03a0 0x7faedc15f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc0b2800 con 0x7faedc15f700 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7faed4001b80 con 0x7faedc15f700 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 >> v1:172.21.15.113:6789/0 conn(0x7faedc15f700 legacy=0x7faedc09aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec4003a00 0x7faec4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc0cffc0 0x7faec4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faec4004d90 0x7faec4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc15fb90 con 0x7faec4003a00 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0b2520 con 0x7faec4004d90 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec400ba10 con 0x7faedc0cffc0 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faedb7fe700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec4003a00 0x7faec4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faedaffd700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc0cffc0 0x7faec4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faedbfff700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faec4004d90 0x7faec4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faedb7fe700 1 -- 172.21.15.113:0/1954434082 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc0cffc0 msgr2=0x7faec4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faedb7fe700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc0cffc0 0x7faec4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faedb7fe700 1 -- 172.21.15.113:0/1954434082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faec4004d90 msgr2=0x7faec4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faedb7fe700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faec4004d90 0x7faec4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faedb7fe700 1 -- 172.21.15.113:0/1954434082 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec40035b0 con 0x7faec4003a00 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faedaffd700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc0cffc0 0x7faec4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faedb7fe700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec4003a00 0x7faec4003e20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7faed0009b60 tx=0x7faec4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6157939bf3dae98a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed000e6c0 con 0x7faec4003a00 2024-01-15T11:25:58.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed000e820 con 0x7faec4003a00 2024-01-15T11:25:58.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.076+0000 7faeda7fc700 1 -- 172.21.15.113:0/1954434082 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed000eb90 con 0x7faec4003a00 2024-01-15T11:25:58.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.077+0000 7faee1d6d700 1 -- 172.21.15.113:0/1954434082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec4003a00 msgr2=0x7faec4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.077+0000 7faee1d6d700 1 --2- 172.21.15.113:0/1954434082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec4003a00 0x7faec4003e20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7faed0009b60 tx=0x7faec4005880 comp rx=0 tx=0).stop 2024-01-15T11:25:58.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.077+0000 7faedbfff700 1 -- 172.21.15.113:0/1954434082 reap_dead start 2024-01-15T11:25:58.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.077+0000 7faee1d6d700 1 -- 172.21.15.113:0/1954434082 shutdown_connections 2024-01-15T11:25:58.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.077+0000 7faee1d6d700 1 -- 172.21.15.113:0/1954434082 >> 172.21.15.113:0/1954434082 conn(0x7faedc0d1e50 msgr2=0x7faedc0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:58.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.077+0000 7faee1d6d700 1 -- 172.21.15.113:0/1954434082 shutdown_connections 2024-01-15T11:25:58.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.077+0000 7faee1d6d700 1 -- 172.21.15.113:0/1954434082 wait complete. 2024-01-15T11:25:58.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.079+0000 7faee1d6d700 1 Processor -- start 2024-01-15T11:25:58.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.079+0000 7faee1d6d700 1 -- start start 2024-01-15T11:25:58.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.079+0000 7faee1d6d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faec4003a00 0x7faedc0ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.079+0000 7faee1d6d700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc1f1ba0 0x7faedc0b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.079+0000 7faee1d6d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faedc1f2020 0x7faedc0b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.079+0000 7faee1d6d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc1f2510 con 0x7faedc1f2020 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.079+0000 7faee1d6d700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc1f27f0 con 0x7faec4003a00 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.079+0000 7faee1d6d700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc1f2ad0 con 0x7faedc1f1ba0 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faedbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faedc1f2020 0x7faedc0b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faedbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faedc1f2020 0x7faedc0b0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33424/0 (socket says 172.21.15.113:33424) 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faedbfff700 1 -- 172.21.15.113:0/1465680728 learned_addr learned my addr 172.21.15.113:0/1465680728 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faedb7fe700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faec4003a00 0x7faedc0ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faedaffd700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc1f1ba0 0x7faedc0b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faedbfff700 1 -- 172.21.15.113:0/1465680728 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc1f1ba0 msgr2=0x7faedc0b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faedbfff700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc1f1ba0 0x7faedc0b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faedbfff700 1 -- 172.21.15.113:0/1465680728 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faec4003a00 msgr2=0x7faedc0ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faedbfff700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faec4003a00 0x7faedc0ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faedbfff700 1 -- 172.21.15.113:0/1465680728 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc0ca010 con 0x7faedc1f2020 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faedbfff700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faedc1f2020 0x7faedc0b0f20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faed4000f80 tx=0x7faed400e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faec3fff700 1 -- 172.21.15.113:0/1465680728 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed400ebc0 con 0x7faedc1f2020 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faec3fff700 1 -- 172.21.15.113:0/1465680728 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed400ed20 con 0x7faedc1f2020 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faee1d6d700 1 -- 172.21.15.113:0/1465680728 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faedc0ca2f0 con 0x7faedc1f2020 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faec3fff700 1 -- 172.21.15.113:0/1465680728 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4013660 con 0x7faedc1f2020 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.080+0000 7faee1d6d700 1 -- 172.21.15.113:0/1465680728 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faedc0caf30 con 0x7faedc1f2020 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.081+0000 7faee1d6d700 1 -- 172.21.15.113:0/1465680728 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faedc069020 con 0x7faedc1f2020 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.082+0000 7faec3fff700 1 -- 172.21.15.113:0/1465680728 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7faed40137c0 con 0x7faedc1f2020 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.082+0000 7faec3fff700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7faec806b350 0x7faec806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.082+0000 7faec3fff700 1 -- 172.21.15.113:0/1465680728 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faed4089cf0 con 0x7faedc1f2020 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.082+0000 7faedb7fe700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7faec806b350 0x7faec806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.082+0000 7faedb7fe700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7faec806b350 0x7faec806d810 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faed00090b0 tx=0x7faed0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:58.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.086+0000 7faec3fff700 1 -- 172.21.15.113:0/1465680728 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faed408e070 con 0x7faedc1f2020 2024-01-15T11:25:58.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.277+0000 7faee1d6d700 1 -- 172.21.15.113:0/1465680728 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faedc0b3860 con 0x7faedc1f2020 2024-01-15T11:25:58.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.278+0000 7faec3fff700 1 -- 172.21.15.113:0/1465680728 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7faedc0b3860 con 0x7faedc1f2020 2024-01-15T11:25:58.277 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-15T11:25:58.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.280+0000 7faee1d6d700 1 -- 172.21.15.113:0/1465680728 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7faec806b350 msgr2=0x7faec806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.280+0000 7faee1d6d700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7faec806b350 0x7faec806d810 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faed00090b0 tx=0x7faed0008030 comp rx=0 tx=0).stop 2024-01-15T11:25:58.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.280+0000 7faee1d6d700 1 -- 172.21.15.113:0/1465680728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faedc1f2020 msgr2=0x7faedc0b0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.280+0000 7faee1d6d700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faedc1f2020 0x7faedc0b0f20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faed4000f80 tx=0x7faed400e440 comp rx=0 tx=0).stop 2024-01-15T11:25:58.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.280+0000 7faee1d6d700 1 -- 172.21.15.113:0/1465680728 shutdown_connections 2024-01-15T11:25:58.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.280+0000 7faee1d6d700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7faec806b350 0x7faec806d810 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.280+0000 7faee1d6d700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faedc1f2020 0x7faedc0b0f20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.280+0000 7faee1d6d700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faec4003a00 0x7faedc0ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.280+0000 7faee1d6d700 1 --2- 172.21.15.113:0/1465680728 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7faedc1f1ba0 0x7faedc0b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.280+0000 7faee1d6d700 1 -- 172.21.15.113:0/1465680728 >> 172.21.15.113:0/1465680728 conn(0x7faedc0d1e50 msgr2=0x7faedc0cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:58.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.280+0000 7faee1d6d700 1 -- 172.21.15.113:0/1465680728 shutdown_connections 2024-01-15T11:25:58.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.280+0000 7faee1d6d700 1 -- 172.21.15.113:0/1465680728 wait complete. 2024-01-15T11:25:58.291 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T11:25:58.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.413+0000 7f485b4e8700 1 Processor -- start 2024-01-15T11:25:58.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.413+0000 7f485b4e8700 1 -- start start 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.414+0000 7f485b4e8700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f485415e120 0x7f485406edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.414+0000 7f485b4e8700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f485406b720 0x7f48540d1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.414+0000 7f485b4e8700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48540cfd80 con 0x7f48540d1ff0 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.414+0000 7f485b4e8700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48540d2480 con 0x7f485406b720 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.414+0000 7f485b4e8700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485415e500 con 0x7f485415e120 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.414+0000 7f4859284700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f485415e120 0x7f485406edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.414+0000 7f4858a83700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f485406b720 0x7f48540d1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.414+0000 7f4859a85700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f48540d1ff0 0x7f4854161b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:60584/0 (socket says 172.21.15.113:60584) 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.414+0000 7f4859a85700 1 -- 172.21.15.113:0/946434386 learned_addr learned my addr 172.21.15.113:0/946434386 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.414+0000 7f4858a83700 1 -- 172.21.15.113:0/946434386 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f485415e120 msgr2=0x7f485406edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.414+0000 7f4858a83700 1 --2- 172.21.15.113:0/946434386 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f485415e120 0x7f485406edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f4858a83700 1 -- 172.21.15.113:0/946434386 >> v1:172.21.15.141:6789/0 conn(0x7f48540d1ff0 legacy=0x7f4854161b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f4858a83700 1 -- 172.21.15.113:0/946434386 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4854070080 con 0x7f485406b720 2024-01-15T11:25:58.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f4859284700 1 --2- 172.21.15.113:0/946434386 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f485415e120 0x7f485406edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:58.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f4858a83700 1 --2- 172.21.15.113:0/946434386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f485406b720 0x7f48540d1ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4850009870 tx=0x7f4850009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=934bd83e6396e033 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:58.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f484bfff700 1 -- 172.21.15.113:0/946434386 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4850011830 con 0x7f485406b720 2024-01-15T11:25:58.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f484bfff700 1 -- 172.21.15.113:0/946434386 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4850011990 con 0x7f485406b720 2024-01-15T11:25:58.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f484bfff700 1 -- 172.21.15.113:0/946434386 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4850011d20 con 0x7f485406b720 2024-01-15T11:25:58.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f485b4e8700 1 -- 172.21.15.113:0/946434386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f485406b720 msgr2=0x7f48540d1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f485b4e8700 1 --2- 172.21.15.113:0/946434386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f485406b720 0x7f48540d1ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4850009870 tx=0x7f4850009b80 comp rx=0 tx=0).stop 2024-01-15T11:25:58.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f485b4e8700 1 -- 172.21.15.113:0/946434386 shutdown_connections 2024-01-15T11:25:58.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f485b4e8700 1 --2- 172.21.15.113:0/946434386 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f485415e120 0x7f485406edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f485b4e8700 1 --2- 172.21.15.113:0/946434386 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f485406b720 0x7f48540d1ab0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.415+0000 7f485b4e8700 1 -- 172.21.15.113:0/946434386 >> 172.21.15.113:0/946434386 conn(0x7f4854073270 msgr2=0x7f485409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.416+0000 7f485b4e8700 1 -- 172.21.15.113:0/946434386 shutdown_connections 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.416+0000 7f485b4e8700 1 -- 172.21.15.113:0/946434386 wait complete. 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.417+0000 7f485b4e8700 1 Processor -- start 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.417+0000 7f485b4e8700 1 -- start start 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.417+0000 7f485b4e8700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f485406b720 0x7f48540aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.417+0000 7f485b4e8700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f48540d1ff0 0x7f48540af480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.417+0000 7f485b4e8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f485415e120 0x7f48540c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.417+0000 7f485b4e8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485409e570 con 0x7f485415e120 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.417+0000 7f485b4e8700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48540af9c0 con 0x7f48540d1ff0 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.417+0000 7f485b4e8700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48540afc60 con 0x7f485406b720 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4859a85700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f485415e120 0x7f48540c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4859a85700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f485415e120 0x7f48540c7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33440/0 (socket says 172.21.15.113:33440) 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4859a85700 1 -- 172.21.15.113:0/3562198216 learned_addr learned my addr 172.21.15.113:0/3562198216 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4859284700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f485406b720 0x7f48540aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4858a83700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f48540d1ff0 0x7f48540af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4859a85700 1 -- 172.21.15.113:0/3562198216 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f485406b720 msgr2=0x7f48540aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4859a85700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f485406b720 0x7f48540aed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4859a85700 1 -- 172.21.15.113:0/3562198216 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f48540d1ff0 msgr2=0x7f48540af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4859a85700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f48540d1ff0 0x7f48540af480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4859a85700 1 -- 172.21.15.113:0/3562198216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48440094a0 con 0x7f485415e120 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4859a85700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f485415e120 0x7f48540c7aa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f484c00cbc0 tx=0x7f484c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4849ffb700 1 -- 172.21.15.113:0/3562198216 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c00ab50 con 0x7f485415e120 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4849ffb700 1 -- 172.21.15.113:0/3562198216 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c00acb0 con 0x7f485415e120 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f485b4e8700 1 -- 172.21.15.113:0/3562198216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48500094a0 con 0x7f485415e120 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f4849ffb700 1 -- 172.21.15.113:0/3562198216 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c012600 con 0x7f485415e120 2024-01-15T11:25:58.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.418+0000 7f485b4e8700 1 -- 172.21.15.113:0/3562198216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485409eaf0 con 0x7f485415e120 2024-01-15T11:25:58.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.419+0000 7f485b4e8700 1 -- 172.21.15.113:0/3562198216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4854068b00 con 0x7f485415e120 2024-01-15T11:25:58.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.420+0000 7f4849ffb700 1 -- 172.21.15.113:0/3562198216 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f484c01b3b0 con 0x7f485415e120 2024-01-15T11:25:58.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.420+0000 7f4849ffb700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f483806b350 0x7f483806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.420+0000 7f4859284700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f483806b350 0x7f483806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.420+0000 7f4849ffb700 1 -- 172.21.15.113:0/3562198216 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f484c089850 con 0x7f485415e120 2024-01-15T11:25:58.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.421+0000 7f4859284700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f483806b350 0x7f483806d810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4844009c60 tx=0x7f4844008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:58.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.425+0000 7f4849ffb700 1 -- 172.21.15.113:0/3562198216 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f484c0546f0 con 0x7f485415e120 2024-01-15T11:25:58.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.659+0000 7f485b4e8700 1 -- 172.21.15.113:0/3562198216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4854067f60 con 0x7f485415e120 2024-01-15T11:25:58.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.659+0000 7f4849ffb700 1 -- 172.21.15.113:0/3562198216 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f484c017090 con 0x7f485415e120 2024-01-15T11:25:58.658 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:58.658 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3164513953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":3164513953},{"type":"v1","addr":"172.21.15.113:6835","nonce":3164513953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/891522731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":891522731},{"type":"v1","addr":"172.21.15.113:6837","nonce":891522731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2158560535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2158560535},{"type":"v1","addr":"172.21.15.141:6834","nonce":2158560535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T11:25:15.611582+0000","modified":"2024-01-15T11:25:18.718623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4300":{"gid":4300,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1012738257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1012738257},{"type":"v1","addr":"172.21.15.141:6835","nonce":1012738257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T11:25:58.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.661+0000 7f485b4e8700 1 -- 172.21.15.113:0/3562198216 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f483806b350 msgr2=0x7f483806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.661+0000 7f485b4e8700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f483806b350 0x7f483806d810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4844009c60 tx=0x7f4844008040 comp rx=0 tx=0).stop 2024-01-15T11:25:58.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.661+0000 7f485b4e8700 1 -- 172.21.15.113:0/3562198216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f485415e120 msgr2=0x7f48540c7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.661+0000 7f485b4e8700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f485415e120 0x7f48540c7aa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f484c00cbc0 tx=0x7f484c00cf80 comp rx=0 tx=0).stop 2024-01-15T11:25:58.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.661+0000 7f485b4e8700 1 -- 172.21.15.113:0/3562198216 shutdown_connections 2024-01-15T11:25:58.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.661+0000 7f485b4e8700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f483806b350 0x7f483806d810 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.661+0000 7f485b4e8700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f485415e120 0x7f48540c7aa0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.661+0000 7f485b4e8700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f485406b720 0x7f48540aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.661+0000 7f485b4e8700 1 --2- 172.21.15.113:0/3562198216 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f48540d1ff0 0x7f48540af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.661+0000 7f485b4e8700 1 -- 172.21.15.113:0/3562198216 >> 172.21.15.113:0/3562198216 conn(0x7f4854073270 msgr2=0x7f48540b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:58.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.661+0000 7f485b4e8700 1 -- 172.21.15.113:0/3562198216 shutdown_connections 2024-01-15T11:25:58.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.661+0000 7f485b4e8700 1 -- 172.21.15.113:0/3562198216 wait complete. 2024-01-15T11:25:58.662 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-15T11:25:58.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:58.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.794+0000 7f45b79d0700 1 Processor -- start 2024-01-15T11:25:58.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.794+0000 7f45b79d0700 1 -- start start 2024-01-15T11:25:58.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45b79d0700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f45b00ce090 0x7f45b006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45b79d0700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b006e7f0 con 0x7f45b015e2a0 2024-01-15T11:25:58.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45b79d0700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b006dec0 con 0x7f45b0073f90 2024-01-15T11:25:58.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45b79d0700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b006e1a0 con 0x7f45b00ce090 2024-01-15T11:25:58.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45b576c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f45b015e2a0 0x7f45b006ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48340/0 (socket says 172.21.15.113:48340) 2024-01-15T11:25:58.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45b576c700 1 -- 172.21.15.113:0/4181292886 learned_addr learned my addr 172.21.15.113:0/4181292886 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:58.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45b4f6b700 1 --2- 172.21.15.113:0/4181292886 >> v2:172.21.15.113:3300/0 conn(0x7f45b00ce090 0x7f45b006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45a7fff700 1 -- 172.21.15.113:0/4181292886 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361388688 0 0) 0x7f45b006e7f0 con 0x7f45b015e2a0 2024-01-15T11:25:58.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45a7fff700 1 -- 172.21.15.113:0/4181292886 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f459c0035b0 con 0x7f45b015e2a0 2024-01-15T11:25:58.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45b4f6b700 1 -- 172.21.15.113:0/4181292886 >> v1:172.21.15.141:6789/0 conn(0x7f45b0073f90 legacy=0x7f45b00cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45b4f6b700 1 -- 172.21.15.113:0/4181292886 >> v1:172.21.15.113:6789/0 conn(0x7f45b015e2a0 legacy=0x7f45b006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45b4f6b700 1 -- 172.21.15.113:0/4181292886 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b006e480 con 0x7f45b00ce090 2024-01-15T11:25:58.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.795+0000 7f45b4f6b700 1 --2- 172.21.15.113:0/4181292886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b00ce090 0x7f45b006ec80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009840 tx=0x7f45a0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f2c2c469d93b8d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:58.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.796+0000 7f45a7fff700 1 -- 172.21.15.113:0/4181292886 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0011830 con 0x7f45b00ce090 2024-01-15T11:25:58.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.796+0000 7f45a7fff700 1 -- 172.21.15.113:0/4181292886 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a0011990 con 0x7f45b00ce090 2024-01-15T11:25:58.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.796+0000 7f45a7fff700 1 -- 172.21.15.113:0/4181292886 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0011d20 con 0x7f45b00ce090 2024-01-15T11:25:58.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.796+0000 7f45b79d0700 1 -- 172.21.15.113:0/4181292886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b00ce090 msgr2=0x7f45b006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.796+0000 7f45b79d0700 1 --2- 172.21.15.113:0/4181292886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b00ce090 0x7f45b006ec80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009840 tx=0x7f45a0009b50 comp rx=0 tx=0).stop 2024-01-15T11:25:58.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.796+0000 7f45b79d0700 1 -- 172.21.15.113:0/4181292886 shutdown_connections 2024-01-15T11:25:58.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.796+0000 7f45b79d0700 1 --2- 172.21.15.113:0/4181292886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b00ce090 0x7f45b006ec80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.796+0000 7f45b79d0700 1 -- 172.21.15.113:0/4181292886 >> 172.21.15.113:0/4181292886 conn(0x7f45b00cd570 msgr2=0x7f45b00cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:58.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.796+0000 7f45b79d0700 1 -- 172.21.15.113:0/4181292886 shutdown_connections 2024-01-15T11:25:58.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.796+0000 7f45b79d0700 1 -- 172.21.15.113:0/4181292886 wait complete. 2024-01-15T11:25:58.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.797+0000 7f45b79d0700 1 Processor -- start 2024-01-15T11:25:58.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.797+0000 7f45b79d0700 1 -- start start 2024-01-15T11:25:58.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b79d0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b0073f90 0x7f45b0157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b79d0700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45b00ce090 0x7f45b01563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b79d0700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f45b015e2a0 0x7f45b01f4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b79d0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b01590d0 con 0x7f45b0073f90 2024-01-15T11:25:58.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b79d0700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b0158270 con 0x7f45b00ce090 2024-01-15T11:25:58.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b79d0700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b0156910 con 0x7f45b015e2a0 2024-01-15T11:25:58.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b576c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b0073f90 0x7f45b0157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b576c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b0073f90 0x7f45b0157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33462/0 (socket says 172.21.15.113:33462) 2024-01-15T11:25:58.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b576c700 1 -- 172.21.15.113:0/489938058 learned_addr learned my addr 172.21.15.113:0/489938058 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:58.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b576c700 1 -- 172.21.15.113:0/489938058 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f45b015e2a0 msgr2=0x7f45b01f4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b576c700 1 --2- 172.21.15.113:0/489938058 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f45b015e2a0 0x7f45b01f4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b576c700 1 -- 172.21.15.113:0/489938058 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45b00ce090 msgr2=0x7f45b01563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b576c700 1 --2- 172.21.15.113:0/489938058 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45b00ce090 0x7f45b01563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.798+0000 7f45b576c700 1 -- 172.21.15.113:0/489938058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a00094a0 con 0x7f45b0073f90 2024-01-15T11:25:58.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.799+0000 7f45b576c700 1 --2- 172.21.15.113:0/489938058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b0073f90 0x7f45b0157d30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f45ac001eb0 tx=0x7f45ac00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:58.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.799+0000 7f45a5ffb700 1 -- 172.21.15.113:0/489938058 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac005d70 con 0x7f45b0073f90 2024-01-15T11:25:58.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.799+0000 7f45a5ffb700 1 -- 172.21.15.113:0/489938058 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac0070d0 con 0x7f45b0073f90 2024-01-15T11:25:58.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.799+0000 7f45b79d0700 1 -- 172.21.15.113:0/489938058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b01f4ad0 con 0x7f45b0073f90 2024-01-15T11:25:58.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.799+0000 7f45a5ffb700 1 -- 172.21.15.113:0/489938058 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac007460 con 0x7f45b0073f90 2024-01-15T11:25:58.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.799+0000 7f45b79d0700 1 -- 172.21.15.113:0/489938058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b0158730 con 0x7f45b0073f90 2024-01-15T11:25:58.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.800+0000 7f45b79d0700 1 -- 172.21.15.113:0/489938058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b0159210 con 0x7f45b0073f90 2024-01-15T11:25:58.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.800+0000 7f45a5ffb700 1 -- 172.21.15.113:0/489938058 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f45ac0075c0 con 0x7f45b0073f90 2024-01-15T11:25:58.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.801+0000 7f45a5ffb700 1 --2- 172.21.15.113:0/489938058 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f459406b350 0x7f459406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:58.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.801+0000 7f45a5ffb700 1 -- 172.21.15.113:0/489938058 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f45ac089060 con 0x7f45b0073f90 2024-01-15T11:25:58.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.801+0000 7f45b4f6b700 1 --2- 172.21.15.113:0/489938058 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f459406b350 0x7f459406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:58.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.801+0000 7f45b4f6b700 1 --2- 172.21.15.113:0/489938058 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f459406b350 0x7f459406d810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f45b015d070 tx=0x7f45a0010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:58.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.805+0000 7f45a5ffb700 1 -- 172.21.15.113:0/489938058 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45b0159210 con 0x7f45b0073f90 2024-01-15T11:25:58.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.996+0000 7f45b79d0700 1 -- 172.21.15.113:0/489938058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45b0159210 con 0x7f45b0073f90 2024-01-15T11:25:58.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.997+0000 7f45a5ffb700 1 -- 172.21.15.113:0/489938058 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16757 (secure 0 0 0) 0x7f45b0159210 con 0x7f45b0073f90 2024-01-15T11:25:58.996 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:58.997 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":22,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:57.253831+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":1}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:25:58.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.999+0000 7f45b79d0700 1 -- 172.21.15.113:0/489938058 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f459406b350 msgr2=0x7f459406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.999+0000 7f45b79d0700 1 --2- 172.21.15.113:0/489938058 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f459406b350 0x7f459406d810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f45b015d070 tx=0x7f45a0010000 comp rx=0 tx=0).stop 2024-01-15T11:25:58.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.999+0000 7f45b79d0700 1 -- 172.21.15.113:0/489938058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b0073f90 msgr2=0x7f45b0157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:58.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.999+0000 7f45b79d0700 1 --2- 172.21.15.113:0/489938058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b0073f90 0x7f45b0157d30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f45ac001eb0 tx=0x7f45ac00c860 comp rx=0 tx=0).stop 2024-01-15T11:25:58.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.999+0000 7f45b79d0700 1 -- 172.21.15.113:0/489938058 shutdown_connections 2024-01-15T11:25:58.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.999+0000 7f45b79d0700 1 --2- 172.21.15.113:0/489938058 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f459406b350 0x7f459406d810 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.999+0000 7f45b79d0700 1 --2- 172.21.15.113:0/489938058 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f45b015e2a0 0x7f45b01f4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.999+0000 7f45b79d0700 1 --2- 172.21.15.113:0/489938058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b0073f90 0x7f45b0157d30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.999+0000 7f45b79d0700 1 --2- 172.21.15.113:0/489938058 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f45b00ce090 0x7f45b01563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:58.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.999+0000 7f45b79d0700 1 -- 172.21.15.113:0/489938058 >> 172.21.15.113:0/489938058 conn(0x7f45b00cd570 msgr2=0x7f45b00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:59.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.999+0000 7f45b79d0700 1 -- 172.21.15.113:0/489938058 shutdown_connections 2024-01-15T11:25:59.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:58.999+0000 7f45b79d0700 1 -- 172.21.15.113:0/489938058 wait complete. 2024-01-15T11:25:59.011 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-15T11:25:59.011 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T11:25:59.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.133+0000 7fedb2609700 1 Processor -- start 2024-01-15T11:25:59.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.133+0000 7fedb2609700 1 -- start start 2024-01-15T11:25:59.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.134+0000 7fedb2609700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedac15ac50 0x7fedac06c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.134+0000 7fedb2609700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fedac0b50c0 0x7fedac0cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.134+0000 7fedb2609700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac06bf50 con 0x7fedac06f460 2024-01-15T11:25:59.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.134+0000 7fedb2609700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedac15b030 con 0x7fedac0b50c0 2024-01-15T11:25:59.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.134+0000 7fedb2609700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedac06e810 con 0x7fedac15ac50 2024-01-15T11:25:59.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.135+0000 7fedabfff700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedac15ac50 0x7fedac06c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.135+0000 7fedb0ba6700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fedac0b50c0 0x7fedac0cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.135+0000 7fedab7fe700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fedac06f460 0x7fedac0b49b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:60604/0 (socket says 172.21.15.113:60604) 2024-01-15T11:25:59.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.135+0000 7fedab7fe700 1 -- 172.21.15.113:0/2626463332 learned_addr learned my addr 172.21.15.113:0/2626463332 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:59.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.135+0000 7fedabfff700 1 -- 172.21.15.113:0/2626463332 >> v2:172.21.15.141:3300/0 conn(0x7fedac0b50c0 msgr2=0x7fedac0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:59.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.135+0000 7fedabfff700 1 --2- 172.21.15.113:0/2626463332 >> v2:172.21.15.141:3300/0 conn(0x7fedac0b50c0 0x7fedac0cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.135+0000 7fedabfff700 1 -- 172.21.15.113:0/2626463332 >> v1:172.21.15.141:6789/0 conn(0x7fedac06f460 legacy=0x7fedac0b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.135+0000 7fedabfff700 1 -- 172.21.15.113:0/2626463332 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac06b340 con 0x7fedac15ac50 2024-01-15T11:25:59.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.135+0000 7fedabfff700 1 --2- 172.21.15.113:0/2626463332 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedac15ac50 0x7fedac06c3e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009870 tx=0x7fed9c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbef5d9f101cdf0c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:59.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.135+0000 7fedaaffd700 1 -- 172.21.15.113:0/2626463332 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c011830 con 0x7fedac15ac50 2024-01-15T11:25:59.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.135+0000 7fedaaffd700 1 -- 172.21.15.113:0/2626463332 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9c011990 con 0x7fedac15ac50 2024-01-15T11:25:59.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.135+0000 7fedaaffd700 1 -- 172.21.15.113:0/2626463332 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c011d20 con 0x7fedac15ac50 2024-01-15T11:25:59.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.136+0000 7fedb2609700 1 -- 172.21.15.113:0/2626463332 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedac15ac50 msgr2=0x7fedac06c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.136+0000 7fedb2609700 1 --2- 172.21.15.113:0/2626463332 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedac15ac50 0x7fedac06c3e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009870 tx=0x7fed9c009b80 comp rx=0 tx=0).stop 2024-01-15T11:25:59.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.136+0000 7fedb2609700 1 -- 172.21.15.113:0/2626463332 shutdown_connections 2024-01-15T11:25:59.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.136+0000 7fedb2609700 1 --2- 172.21.15.113:0/2626463332 >> v2:172.21.15.141:3300/0 conn(0x7fedac0b50c0 0x7fedac0cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.136+0000 7fedb2609700 1 --2- 172.21.15.113:0/2626463332 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedac15ac50 0x7fedac06c3e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.136+0000 7fedb2609700 1 -- 172.21.15.113:0/2626463332 >> 172.21.15.113:0/2626463332 conn(0x7fedac0b3130 msgr2=0x7fedac0d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:59.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.136+0000 7fedb2609700 1 -- 172.21.15.113:0/2626463332 shutdown_connections 2024-01-15T11:25:59.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.136+0000 7fedb2609700 1 -- 172.21.15.113:0/2626463332 wait complete. 2024-01-15T11:25:59.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.137+0000 7fedb2609700 1 Processor -- start 2024-01-15T11:25:59.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.138+0000 7fedb2609700 1 -- start start 2024-01-15T11:25:59.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.138+0000 7fedb2609700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac06f460 0x7fedac14b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedb2609700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedac0b50c0 0x7fedac14da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedb2609700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedac15ac50 0x7fedac15d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedb2609700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac14bc50 con 0x7fedac06f460 2024-01-15T11:25:59.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedb2609700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac14f440 con 0x7fedac15ac50 2024-01-15T11:25:59.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedb2609700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedac14e3a0 con 0x7fedac0b50c0 2024-01-15T11:25:59.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedabfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac06f460 0x7fedac14b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedabfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac06f460 0x7fedac14b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33466/0 (socket says 172.21.15.113:33466) 2024-01-15T11:25:59.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedabfff700 1 -- 172.21.15.113:0/3598470907 learned_addr learned my addr 172.21.15.113:0/3598470907 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:59.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedab7fe700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedac0b50c0 0x7fedac14da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedb0ba6700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedac15ac50 0x7fedac15d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedabfff700 1 -- 172.21.15.113:0/3598470907 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedac0b50c0 msgr2=0x7fedac14da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedabfff700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedac0b50c0 0x7fedac14da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedabfff700 1 -- 172.21.15.113:0/3598470907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedac15ac50 msgr2=0x7fedac15d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedabfff700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedac15ac50 0x7fedac15d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedabfff700 1 -- 172.21.15.113:0/3598470907 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9c0094a0 con 0x7fedac06f460 2024-01-15T11:25:59.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedb0ba6700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedac15ac50 0x7fedac15d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:25:59.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedabfff700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac06f460 0x7fedac14b710 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009840 tx=0x7fed9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:59.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7feda8ff9700 1 -- 172.21.15.113:0/3598470907 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c0036a0 con 0x7fedac06f460 2024-01-15T11:25:59.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7feda8ff9700 1 -- 172.21.15.113:0/3598470907 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9c003800 con 0x7fedac06f460 2024-01-15T11:25:59.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedb2609700 1 -- 172.21.15.113:0/3598470907 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedac14e680 con 0x7fedac06f460 2024-01-15T11:25:59.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7feda8ff9700 1 -- 172.21.15.113:0/3598470907 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c003bb0 con 0x7fedac06f460 2024-01-15T11:25:59.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.139+0000 7fedb2609700 1 -- 172.21.15.113:0/3598470907 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedac14ec40 con 0x7fedac06f460 2024-01-15T11:25:59.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.140+0000 7fedb2609700 1 -- 172.21.15.113:0/3598470907 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedac067f60 con 0x7fedac06f460 2024-01-15T11:25:59.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.141+0000 7feda8ff9700 1 -- 172.21.15.113:0/3598470907 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fed9c003d10 con 0x7fedac06f460 2024-01-15T11:25:59.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.141+0000 7feda8ff9700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fed8c06b350 0x7fed8c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.141+0000 7feda8ff9700 1 -- 172.21.15.113:0/3598470907 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fed9c08cf50 con 0x7fedac06f460 2024-01-15T11:25:59.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.141+0000 7fedab7fe700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fed8c06b350 0x7fed8c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.141+0000 7fedab7fe700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fed8c06b350 0x7fed8c06d810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed94000f60 tx=0x7fed94009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:59.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.145+0000 7feda8ff9700 1 -- 172.21.15.113:0/3598470907 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed9c057dd0 con 0x7fedac06f460 2024-01-15T11:25:59.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.383+0000 7fedb2609700 1 -- 172.21.15.113:0/3598470907 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedac06e350 con 0x7fedac06f460 2024-01-15T11:25:59.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.384+0000 7feda8ff9700 1 -- 172.21.15.113:0/3598470907 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7fed9c015090 con 0x7fedac06f460 2024-01-15T11:25:59.383 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:59.384 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3164513953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":3164513953},{"type":"v1","addr":"172.21.15.113:6835","nonce":3164513953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/891522731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":891522731},{"type":"v1","addr":"172.21.15.113:6837","nonce":891522731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2158560535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2158560535},{"type":"v1","addr":"172.21.15.141:6834","nonce":2158560535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T11:25:15.611582+0000","modified":"2024-01-15T11:25:18.718623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4300":{"gid":4300,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1012738257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1012738257},{"type":"v1","addr":"172.21.15.141:6835","nonce":1012738257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T11:25:59.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.386+0000 7fedb2609700 1 -- 172.21.15.113:0/3598470907 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fed8c06b350 msgr2=0x7fed8c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.386+0000 7fedb2609700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fed8c06b350 0x7fed8c06d810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed94000f60 tx=0x7fed94009040 comp rx=0 tx=0).stop 2024-01-15T11:25:59.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.386+0000 7fedb2609700 1 -- 172.21.15.113:0/3598470907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac06f460 msgr2=0x7fedac14b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.386+0000 7fedb2609700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac06f460 0x7fedac14b710 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009840 tx=0x7fed9c004750 comp rx=0 tx=0).stop 2024-01-15T11:25:59.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.386+0000 7fedb2609700 1 -- 172.21.15.113:0/3598470907 shutdown_connections 2024-01-15T11:25:59.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.386+0000 7fedb2609700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fed8c06b350 0x7fed8c06d810 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.386+0000 7fedb2609700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedac15ac50 0x7fedac15d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.386+0000 7fedb2609700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac06f460 0x7fedac14b710 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.386+0000 7fedb2609700 1 --2- 172.21.15.113:0/3598470907 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedac0b50c0 0x7fedac14da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.386+0000 7fedb2609700 1 -- 172.21.15.113:0/3598470907 >> 172.21.15.113:0/3598470907 conn(0x7fedac0b3130 msgr2=0x7fedac0b3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:59.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.386+0000 7fedb2609700 1 -- 172.21.15.113:0/3598470907 shutdown_connections 2024-01-15T11:25:59.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.386+0000 7fedb2609700 1 -- 172.21.15.113:0/3598470907 wait complete. 2024-01-15T11:25:59.387 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-15T11:25:59.400 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T11:25:59.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.523+0000 7ff2e6074700 1 Processor -- start 2024-01-15T11:25:59.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.523+0000 7ff2e6074700 1 -- start start 2024-01-15T11:25:59.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.523+0000 7ff2e6074700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2e006ee50 0x7ff2e0074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2e6074700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e0074e90 con 0x7ff2e0073f90 2024-01-15T11:25:59.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2e6074700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e00748b0 con 0x7ff2e015c140 2024-01-15T11:25:59.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2e6074700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e015c700 con 0x7ff2e006ee50 2024-01-15T11:25:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2df7fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ff2e0073f90 0x7ff2e00ce090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48350/0 (socket says 172.21.15.113:48350) 2024-01-15T11:25:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2df7fe700 1 -- 172.21.15.113:0/2863819002 learned_addr learned my addr 172.21.15.113:0/2863819002 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2deffd700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2e006ee50 0x7ff2e0074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709015795 0 0) 0x7ff2e0074e90 con 0x7ff2e0073f90 2024-01-15T11:25:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2cc0035b0 con 0x7ff2e0073f90 2024-01-15T11:25:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022415972 0 0) 0x7ff2e00748b0 con 0x7ff2e015c140 2024-01-15T11:25:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e0074e90 con 0x7ff2e015c140 2024-01-15T11:25:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3943884185 0 0) 0x7ff2cc0035b0 con 0x7ff2e0073f90 2024-01-15T11:25:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2e006ee50 msgr2=0x7ff2e0074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2e006ee50 0x7ff2e0074370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 >> v1:172.21.15.141:6789/0 conn(0x7ff2e015c140 legacy=0x7ff2e00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e015c9e0 con 0x7ff2e0073f90 2024-01-15T11:25:59.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2deffd700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2e006ee50 0x7ff2e0074370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:25:59.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7ff2c80015f0 con 0x7ff2e0073f90 2024-01-15T11:25:59.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 >> v1:172.21.15.113:6789/0 conn(0x7ff2e0073f90 legacy=0x7ff2e00ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2cc003bc0 0x7ff2cc003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2e006ee50 0x7ff2cc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2cc004d90 0x7ff2cc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e015c700 con 0x7ff2cc003bc0 2024-01-15T11:25:59.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2cc00bb20 con 0x7ff2cc004d90 2024-01-15T11:25:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2cc00bce0 con 0x7ff2e006ee50 2024-01-15T11:25:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.524+0000 7ff2df7fe700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2cc003bc0 0x7ff2cc003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2deffd700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2e006ee50 0x7ff2cc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2dffff700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2cc004d90 0x7ff2cc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2df7fe700 1 -- 172.21.15.113:0/2863819002 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2e006ee50 msgr2=0x7ff2cc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2df7fe700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2e006ee50 0x7ff2cc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2df7fe700 1 -- 172.21.15.113:0/2863819002 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2cc004d90 msgr2=0x7ff2cc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2df7fe700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2cc004d90 0x7ff2cc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2df7fe700 1 -- 172.21.15.113:0/2863819002 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d00094a0 con 0x7ff2cc003bc0 2024-01-15T11:25:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2df7fe700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2cc003bc0 0x7ff2cc003fc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8007fd0 tx=0x7ff2cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7953ece704ba349 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c800cd10 con 0x7ff2cc003bc0 2024-01-15T11:25:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c80019b0 con 0x7ff2cc003bc0 2024-01-15T11:25:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2de7fc700 1 -- 172.21.15.113:0/2863819002 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8005ba0 con 0x7ff2cc003bc0 2024-01-15T11:25:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2e6074700 1 -- 172.21.15.113:0/2863819002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2cc003bc0 msgr2=0x7ff2cc003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2e6074700 1 --2- 172.21.15.113:0/2863819002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2cc003bc0 0x7ff2cc003fc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8007fd0 tx=0x7ff2cc005880 comp rx=0 tx=0).stop 2024-01-15T11:25:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2dffff700 1 -- 172.21.15.113:0/2863819002 reap_dead start 2024-01-15T11:25:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2e6074700 1 -- 172.21.15.113:0/2863819002 shutdown_connections 2024-01-15T11:25:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2e6074700 1 -- 172.21.15.113:0/2863819002 >> 172.21.15.113:0/2863819002 conn(0x7ff2e00cd570 msgr2=0x7ff2e00cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2e6074700 1 -- 172.21.15.113:0/2863819002 shutdown_connections 2024-01-15T11:25:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.525+0000 7ff2e6074700 1 -- 172.21.15.113:0/2863819002 wait complete. 2024-01-15T11:25:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.527+0000 7ff2e6074700 1 Processor -- start 2024-01-15T11:25:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.527+0000 7ff2e6074700 1 -- start start 2024-01-15T11:25:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2e6074700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2cc003bc0 0x7ff2e00b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2e6074700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2e0150420 0x7ff2e014c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2e6074700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2e00b5480 0x7ff2e0158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2e6074700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e014cca0 con 0x7ff2e0150420 2024-01-15T11:25:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2e6074700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e0150c10 con 0x7ff2e00b5480 2024-01-15T11:25:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2e6074700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e00b43e0 con 0x7ff2cc003bc0 2024-01-15T11:25:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2deffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2e0150420 0x7ff2e014c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2deffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2e0150420 0x7ff2e014c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33482/0 (socket says 172.21.15.113:33482) 2024-01-15T11:25:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2deffd700 1 -- 172.21.15.113:0/3108810157 learned_addr learned my addr 172.21.15.113:0/3108810157 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2dffff700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2e00b5480 0x7ff2e0158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2df7fe700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2cc003bc0 0x7ff2e00b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2deffd700 1 -- 172.21.15.113:0/3108810157 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2cc003bc0 msgr2=0x7ff2e00b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2deffd700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2cc003bc0 0x7ff2e00b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2deffd700 1 -- 172.21.15.113:0/3108810157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2e00b5480 msgr2=0x7ff2e0158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2deffd700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2e00b5480 0x7ff2e0158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2deffd700 1 -- 172.21.15.113:0/3108810157 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e00b46c0 con 0x7ff2e0150420 2024-01-15T11:25:59.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2dffff700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2e00b5480 0x7ff2e0158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:59.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2deffd700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2e0150420 0x7ff2e014c760 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0011900 tx=0x7ff2d0011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:59.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2d7fff700 1 -- 172.21.15.113:0/3108810157 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d000fae0 con 0x7ff2e0150420 2024-01-15T11:25:59.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2d7fff700 1 -- 172.21.15.113:0/3108810157 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d000fc40 con 0x7ff2e0150420 2024-01-15T11:25:59.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2d7fff700 1 -- 172.21.15.113:0/3108810157 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d0011ea0 con 0x7ff2e0150420 2024-01-15T11:25:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2e6074700 1 -- 172.21.15.113:0/3108810157 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e00b49a0 con 0x7ff2e0150420 2024-01-15T11:25:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.528+0000 7ff2e6074700 1 -- 172.21.15.113:0/3108810157 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e00b4f60 con 0x7ff2e0150420 2024-01-15T11:25:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.529+0000 7ff2e6074700 1 -- 172.21.15.113:0/3108810157 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e00b5180 con 0x7ff2e0150420 2024-01-15T11:25:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.530+0000 7ff2d7fff700 1 -- 172.21.15.113:0/3108810157 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff2d0015460 con 0x7ff2e0150420 2024-01-15T11:25:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.530+0000 7ff2d7fff700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff2c006b380 0x7ff2c006d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.530+0000 7ff2d7fff700 1 -- 172.21.15.113:0/3108810157 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff2d008a0c0 con 0x7ff2e0150420 2024-01-15T11:25:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.530+0000 7ff2df7fe700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff2c006b380 0x7ff2c006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.530+0000 7ff2df7fe700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff2c006b380 0x7ff2c006d840 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8007fd0 tx=0x7ff2c8007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.535+0000 7ff2d7fff700 1 -- 172.21.15.113:0/3108810157 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2d0054f40 con 0x7ff2e0150420 2024-01-15T11:25:59.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.767+0000 7ff2e6074700 1 -- 172.21.15.113:0/3108810157 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2e0067f60 con 0x7ff2e0150420 2024-01-15T11:25:59.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.767+0000 7ff2d7fff700 1 -- 172.21.15.113:0/3108810157 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7ff2d0058560 con 0x7ff2e0150420 2024-01-15T11:25:59.767 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:25:59.767 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3164513953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":3164513953},{"type":"v1","addr":"172.21.15.113:6835","nonce":3164513953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/891522731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":891522731},{"type":"v1","addr":"172.21.15.113:6837","nonce":891522731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/2158560535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":2158560535},{"type":"v1","addr":"172.21.15.141:6834","nonce":2158560535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T11:25:15.611582+0000","modified":"2024-01-15T11:25:18.718623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4300},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4300":{"gid":4300,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6835/1012738257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":1012738257},{"type":"v1","addr":"172.21.15.141:6835","nonce":1012738257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T11:25:59.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.769+0000 7ff2e6074700 1 -- 172.21.15.113:0/3108810157 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff2c006b380 msgr2=0x7ff2c006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.769+0000 7ff2e6074700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff2c006b380 0x7ff2c006d840 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8007fd0 tx=0x7ff2c8007000 comp rx=0 tx=0).stop 2024-01-15T11:25:59.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.769+0000 7ff2e6074700 1 -- 172.21.15.113:0/3108810157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2e0150420 msgr2=0x7ff2e014c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.769+0000 7ff2e6074700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2e0150420 0x7ff2e014c760 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0011900 tx=0x7ff2d0011cc0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.769+0000 7ff2e6074700 1 -- 172.21.15.113:0/3108810157 shutdown_connections 2024-01-15T11:25:59.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.769+0000 7ff2e6074700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff2c006b380 0x7ff2c006d840 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.769+0000 7ff2e6074700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff2e00b5480 0x7ff2e0158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.769+0000 7ff2e6074700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff2cc003bc0 0x7ff2e00b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.769+0000 7ff2e6074700 1 --2- 172.21.15.113:0/3108810157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2e0150420 0x7ff2e014c760 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.769+0000 7ff2e6074700 1 -- 172.21.15.113:0/3108810157 >> 172.21.15.113:0/3108810157 conn(0x7ff2e00cd570 msgr2=0x7ff2e00cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:59.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.769+0000 7ff2e6074700 1 -- 172.21.15.113:0/3108810157 shutdown_connections 2024-01-15T11:25:59.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.769+0000 7ff2e6074700 1 -- 172.21.15.113:0/3108810157 wait complete. 2024-01-15T11:25:59.770 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-15T11:25:59.781 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:25:59.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.902+0000 7fd9707dc700 1 Processor -- start 2024-01-15T11:25:59.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.902+0000 7fd9707dc700 1 -- start start 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.903+0000 7fd9707dc700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd9680d2160 0x7fd9680b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.903+0000 7fd9707dc700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9680cd100 con 0x7fd96815bed0 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.903+0000 7fd9707dc700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9680d09a0 con 0x7fd9681603c0 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.903+0000 7fd9707dc700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96815c2b0 con 0x7fd9680d2160 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.903+0000 7fd96e578700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fd96815bed0 0x7fd9680d1a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48364/0 (socket says 172.21.15.113:48364) 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.903+0000 7fd96e578700 1 -- 172.21.15.113:0/3535319949 learned_addr learned my addr 172.21.15.113:0/3535319949 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383422617 0 0) 0x7fd9680cd100 con 0x7fd96815bed0 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96dd77700 1 --2- 172.21.15.113:0/3535319949 >> v2:172.21.15.141:3300/0 conn(0x7fd9680d2160 0x7fd9680b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9540035b0 con 0x7fd96815bed0 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1920191014 0 0) 0x7fd9540035b0 con 0x7fd96815bed0 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 >> v2:172.21.15.141:3300/0 conn(0x7fd9680d2160 msgr2=0x7fd9680b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 --2- 172.21.15.113:0/3535319949 >> v2:172.21.15.141:3300/0 conn(0x7fd9680d2160 0x7fd9680b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 >> v1:172.21.15.141:6789/0 conn(0x7fd9681603c0 legacy=0x7fd9680cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd968160850 con 0x7fd96815bed0 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fd95c001670 con 0x7fd96815bed0 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 >> v1:172.21.15.113:6789/0 conn(0x7fd96815bed0 legacy=0x7fd9680d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 --2- 172.21.15.113:0/3535319949 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd954003a00 0x7fd954003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 --2- 172.21.15.113:0/3535319949 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd954004850 0x7fd954004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 --2- 172.21.15.113:0/3535319949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd954005210 0x7fd95400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96815c2b0 con 0x7fd954005210 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95400c0b0 con 0x7fd954004850 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95400c270 con 0x7fd954003a00 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96ed79700 1 --2- 172.21.15.113:0/3535319949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd954005210 0x7fd95400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96dd77700 1 --2- 172.21.15.113:0/3535319949 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd954004850 0x7fd954004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96e578700 1 --2- 172.21.15.113:0/3535319949 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd954003a00 0x7fd954003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96ed79700 1 -- 172.21.15.113:0/3535319949 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd954003a00 msgr2=0x7fd954003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96ed79700 1 --2- 172.21.15.113:0/3535319949 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd954003a00 0x7fd954003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96ed79700 1 -- 172.21.15.113:0/3535319949 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd954004850 msgr2=0x7fd954004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:25:59.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96ed79700 1 --2- 172.21.15.113:0/3535319949 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd954004850 0x7fd954004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.904+0000 7fd96ed79700 1 -- 172.21.15.113:0/3535319949 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9540035b0 con 0x7fd954005210 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.905+0000 7fd96ed79700 1 -- 172.21.15.113:0/3535319949 reap_dead start 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.905+0000 7fd96ed79700 1 --2- 172.21.15.113:0/3535319949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd954005210 0x7fd95400ba10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd96000c190 tx=0x7fd954005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b05942baa6fb617 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.905+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd960001db0 con 0x7fd954005210 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.905+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96000ccd0 con 0x7fd954005210 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.905+0000 7fd96d576700 1 -- 172.21.15.113:0/3535319949 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd960003c40 con 0x7fd954005210 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.905+0000 7fd9707dc700 1 -- 172.21.15.113:0/3535319949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd954005210 msgr2=0x7fd95400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.905+0000 7fd9707dc700 1 --2- 172.21.15.113:0/3535319949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd954005210 0x7fd95400ba10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd96000c190 tx=0x7fd954005880 comp rx=0 tx=0).stop 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.905+0000 7fd9707dc700 1 -- 172.21.15.113:0/3535319949 shutdown_connections 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.905+0000 7fd9707dc700 1 --2- 172.21.15.113:0/3535319949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd954005210 0x7fd95400ba10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.905+0000 7fd9707dc700 1 -- 172.21.15.113:0/3535319949 >> 172.21.15.113:0/3535319949 conn(0x7fd968073130 msgr2=0x7fd96809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.905+0000 7fd9707dc700 1 -- 172.21.15.113:0/3535319949 shutdown_connections 2024-01-15T11:25:59.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.905+0000 7fd9707dc700 1 -- 172.21.15.113:0/3535319949 wait complete. 2024-01-15T11:25:59.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.907+0000 7fd9707dc700 1 Processor -- start 2024-01-15T11:25:59.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.907+0000 7fd9707dc700 1 -- start start 2024-01-15T11:25:59.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.907+0000 7fd9707dc700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd954005210 0x7fd9680caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.907+0000 7fd9707dc700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9680ca580 0x7fd96814ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.907+0000 7fd9707dc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9680caa00 0x7fd96814f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.907+0000 7fd9707dc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9680c8860 con 0x7fd9680caa00 2024-01-15T11:25:59.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.907+0000 7fd9707dc700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96814f780 con 0x7fd9680ca580 2024-01-15T11:25:59.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.907+0000 7fd9707dc700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96815c8c0 con 0x7fd954005210 2024-01-15T11:25:59.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.907+0000 7fd96ed79700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9680caa00 0x7fd96814f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.907+0000 7fd96ed79700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9680caa00 0x7fd96814f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33502/0 (socket says 172.21.15.113:33502) 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.907+0000 7fd96ed79700 1 -- 172.21.15.113:0/1702081805 learned_addr learned my addr 172.21.15.113:0/1702081805 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd96e578700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd954005210 0x7fd9680caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd96dd77700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9680ca580 0x7fd96814ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd96ed79700 1 -- 172.21.15.113:0/1702081805 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd954005210 msgr2=0x7fd9680caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd96ed79700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd954005210 0x7fd9680caef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd96ed79700 1 -- 172.21.15.113:0/1702081805 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9680ca580 msgr2=0x7fd96814ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd96ed79700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9680ca580 0x7fd96814ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd96ed79700 1 -- 172.21.15.113:0/1702081805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96815c590 con 0x7fd9680caa00 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd96dd77700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9680ca580 0x7fd96814ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd96ed79700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9680caa00 0x7fd96814f240 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd9680cf700 tx=0x7fd9600014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd95affd700 1 -- 172.21.15.113:0/1702081805 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd960003e20 con 0x7fd9680caa00 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd95affd700 1 -- 172.21.15.113:0/1702081805 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd960005070 con 0x7fd9680caa00 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd9707dc700 1 -- 172.21.15.113:0/1702081805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd968159540 con 0x7fd9680caa00 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd95affd700 1 -- 172.21.15.113:0/1702081805 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd960005220 con 0x7fd9680caa00 2024-01-15T11:25:59.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.908+0000 7fd9707dc700 1 -- 172.21.15.113:0/1702081805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9680c8580 con 0x7fd9680caa00 2024-01-15T11:25:59.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.909+0000 7fd9707dc700 1 -- 172.21.15.113:0/1702081805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9680c55f0 con 0x7fd9680caa00 2024-01-15T11:25:59.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.910+0000 7fd95affd700 1 -- 172.21.15.113:0/1702081805 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd960005380 con 0x7fd9680caa00 2024-01-15T11:25:59.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.910+0000 7fd95affd700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd94c06b380 0x7fd94c06d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:25:59.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.910+0000 7fd95affd700 1 -- 172.21.15.113:0/1702081805 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd96008a520 con 0x7fd9680caa00 2024-01-15T11:25:59.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.910+0000 7fd96e578700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd94c06b380 0x7fd94c06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:25:59.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.910+0000 7fd96e578700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd94c06b380 0x7fd94c06d840 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd95c000c00 tx=0x7fd95c001b20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:25:59.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:25:59.914+0000 7fd95affd700 1 -- 172.21.15.113:0/1702081805 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9600553a0 con 0x7fd9680caa00 2024-01-15T11:26:00.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.105+0000 7fd9707dc700 1 -- 172.21.15.113:0/1702081805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd96815c3f0 con 0x7fd9680caa00 2024-01-15T11:26:00.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.107+0000 7fd95affd700 1 -- 172.21.15.113:0/1702081805 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16757 (secure 0 0 0) 0x7fd96815c3f0 con 0x7fd9680caa00 2024-01-15T11:26:00.106 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:00.106 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":22,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:25:57.253831+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:25:13.398220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T11:25:14.589439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":1}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:26:00.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.109+0000 7fd9707dc700 1 -- 172.21.15.113:0/1702081805 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd94c06b380 msgr2=0x7fd94c06d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:00.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.109+0000 7fd9707dc700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd94c06b380 0x7fd94c06d840 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd95c000c00 tx=0x7fd95c001b20 comp rx=0 tx=0).stop 2024-01-15T11:26:00.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.109+0000 7fd9707dc700 1 -- 172.21.15.113:0/1702081805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9680caa00 msgr2=0x7fd96814f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:00.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.109+0000 7fd9707dc700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9680caa00 0x7fd96814f240 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd9680cf700 tx=0x7fd9600014a0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.109+0000 7fd9707dc700 1 -- 172.21.15.113:0/1702081805 shutdown_connections 2024-01-15T11:26:00.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.109+0000 7fd9707dc700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd94c06b380 0x7fd94c06d840 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.109+0000 7fd9707dc700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9680caa00 0x7fd96814f240 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.109+0000 7fd9707dc700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd954005210 0x7fd9680caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.109+0000 7fd9707dc700 1 --2- 172.21.15.113:0/1702081805 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9680ca580 0x7fd96814ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.109+0000 7fd9707dc700 1 -- 172.21.15.113:0/1702081805 >> 172.21.15.113:0/1702081805 conn(0x7fd968073130 msgr2=0x7fd9680b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:00.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.109+0000 7fd9707dc700 1 -- 172.21.15.113:0/1702081805 shutdown_connections 2024-01-15T11:26:00.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.109+0000 7fd9707dc700 1 -- 172.21.15.113:0/1702081805 wait complete. 2024-01-15T11:26:00.121 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-15T11:26:00.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.244+0000 7f25a4a38700 1 Processor -- start 2024-01-15T11:26:00.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.244+0000 7f25a4a38700 1 -- start start 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.245+0000 7f25a4a38700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f25a015e080 0x7f25a00cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.245+0000 7f25a4a38700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f25a00d22f0 0x7f25a00d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.245+0000 7f25a4a38700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a00d0150 con 0x7f25a00ce4d0 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.245+0000 7f25a4a38700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a015e460 con 0x7f25a015e080 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.245+0000 7f25a4a38700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a00cee50 con 0x7f25a00d22f0 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.245+0000 7f259e59c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f25a015e080 0x7f25a00cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.245+0000 7f259e59c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f25a015e080 0x7f25a00cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33504/0 (socket says 172.21.15.113:33504) 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.245+0000 7f259e59c700 1 -- 172.21.15.113:0/1688936809 learned_addr learned my addr 172.21.15.113:0/1688936809 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.245+0000 7f259e59c700 1 -- 172.21.15.113:0/1688936809 >> v2:172.21.15.141:3300/0 conn(0x7f25a00d22f0 msgr2=0x7f25a00d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.245+0000 7f259ed9d700 1 --2- 172.21.15.113:0/1688936809 >> v2:172.21.15.141:3300/0 conn(0x7f25a00d22f0 0x7f25a00d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f259e59c700 1 --2- 172.21.15.113:0/1688936809 >> v2:172.21.15.141:3300/0 conn(0x7f25a00d22f0 0x7f25a00d05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f259e59c700 1 -- 172.21.15.113:0/1688936809 >> v1:172.21.15.141:6789/0 conn(0x7f25a00ce4d0 legacy=0x7f25a00d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:00.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f259e59c700 1 -- 172.21.15.113:0/1688936809 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a00cf130 con 0x7f25a015e080 2024-01-15T11:26:00.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f259e59c700 1 --2- 172.21.15.113:0/1688936809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a015e080 0x7f25a00cc2a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f25a00c8cb0 tx=0x7f2594009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22f347e49878ea server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:00.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f259d59a700 1 -- 172.21.15.113:0/1688936809 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594004230 con 0x7f25a015e080 2024-01-15T11:26:00.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f259d59a700 1 -- 172.21.15.113:0/1688936809 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2594004390 con 0x7f25a015e080 2024-01-15T11:26:00.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f259d59a700 1 -- 172.21.15.113:0/1688936809 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25940036a0 con 0x7f25a015e080 2024-01-15T11:26:00.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f25a4a38700 1 -- 172.21.15.113:0/1688936809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a015e080 msgr2=0x7f25a00cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:00.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f25a4a38700 1 --2- 172.21.15.113:0/1688936809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a015e080 0x7f25a00cc2a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f25a00c8cb0 tx=0x7f2594009bd0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f25a4a38700 1 -- 172.21.15.113:0/1688936809 shutdown_connections 2024-01-15T11:26:00.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f25a4a38700 1 --2- 172.21.15.113:0/1688936809 >> v2:172.21.15.141:3300/0 conn(0x7f25a00d22f0 0x7f25a00d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f25a4a38700 1 --2- 172.21.15.113:0/1688936809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a015e080 0x7f25a00cc2a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f25a4a38700 1 -- 172.21.15.113:0/1688936809 >> 172.21.15.113:0/1688936809 conn(0x7f25a0073130 msgr2=0x7f25a009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:00.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f25a4a38700 1 -- 172.21.15.113:0/1688936809 shutdown_connections 2024-01-15T11:26:00.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.246+0000 7f25a4a38700 1 -- 172.21.15.113:0/1688936809 wait complete. 2024-01-15T11:26:00.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.248+0000 7f25a4a38700 1 Processor -- start 2024-01-15T11:26:00.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.248+0000 7f25a4a38700 1 -- start start 2024-01-15T11:26:00.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.248+0000 7f25a4a38700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25a00ce4d0 0x7f25a00b6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.248+0000 7f25a4a38700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a00d22f0 0x7f25a00ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.248+0000 7f25a4a38700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25a015e080 0x7f25a015c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.248+0000 7f25a4a38700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00c9730 con 0x7f25a00d22f0 2024-01-15T11:26:00.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.248+0000 7f25a4a38700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00c9980 con 0x7f25a015e080 2024-01-15T11:26:00.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.248+0000 7f25a4a38700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00c9c60 con 0x7f25a00ce4d0 2024-01-15T11:26:00.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259dd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a00d22f0 0x7f25a00ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259dd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a00d22f0 0x7f25a00ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33510/0 (socket says 172.21.15.113:33510) 2024-01-15T11:26:00.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259dd9b700 1 -- 172.21.15.113:0/3000929014 learned_addr learned my addr 172.21.15.113:0/3000929014 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259ed9d700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25a015e080 0x7f25a015c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259e59c700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25a00ce4d0 0x7f25a00b6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259dd9b700 1 -- 172.21.15.113:0/3000929014 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25a00ce4d0 msgr2=0x7f25a00b6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259dd9b700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25a00ce4d0 0x7f25a00b6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259dd9b700 1 -- 172.21.15.113:0/3000929014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25a015e080 msgr2=0x7f25a015c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259dd9b700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25a015e080 0x7f25a015c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259dd9b700 1 -- 172.21.15.113:0/3000929014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2594009550 con 0x7f25a00d22f0 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259e59c700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25a00ce4d0 0x7f25a00b6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259ed9d700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25a015e080 0x7f25a015c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f259dd9b700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a00d22f0 0x7f25a00ca740 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f259000b4a0 tx=0x7f259000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f258effd700 1 -- 172.21.15.113:0/3000929014 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2590011830 con 0x7f25a00d22f0 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f258effd700 1 -- 172.21.15.113:0/3000929014 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2590011990 con 0x7f25a00d22f0 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f25a4a38700 1 -- 172.21.15.113:0/3000929014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a00c9f40 con 0x7f25a00d22f0 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f258effd700 1 -- 172.21.15.113:0/3000929014 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2590011d40 con 0x7f25a00d22f0 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.249+0000 7f25a4a38700 1 -- 172.21.15.113:0/3000929014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a00b6c80 con 0x7f25a00d22f0 2024-01-15T11:26:00.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.250+0000 7f25a4a38700 1 -- 172.21.15.113:0/3000929014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a00ca460 con 0x7f25a00d22f0 2024-01-15T11:26:00.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.251+0000 7f258effd700 1 -- 172.21.15.113:0/3000929014 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f259001a400 con 0x7f25a00d22f0 2024-01-15T11:26:00.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.251+0000 7f258effd700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f258006b350 0x7f258006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.251+0000 7f259e59c700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f258006b350 0x7f258006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.251+0000 7f258effd700 1 -- 172.21.15.113:0/3000929014 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f25900898a0 con 0x7f25a00d22f0 2024-01-15T11:26:00.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.252+0000 7f259e59c700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f258006b350 0x7f258006d810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f25a009b260 tx=0x7f2594008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:00.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.256+0000 7f258effd700 1 -- 172.21.15.113:0/3000929014 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25a00ca460 con 0x7f25a00d22f0 2024-01-15T11:26:00.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.491+0000 7f25a4a38700 1 -- 172.21.15.113:0/3000929014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f25a00ca460 con 0x7f25a00d22f0 2024-01-15T11:26:00.491 INFO:tasks.ceph.mon.a.smithi113.stderr:2024-01-15T11:26:00.492+0000 7f80ebd4a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T11:26:00.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.496+0000 7f258effd700 1 -- 172.21.15.113:0/3000929014 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f25a00ca460 con 0x7f25a00d22f0 2024-01-15T11:26:00.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.499+0000 7f25a4a38700 1 -- 172.21.15.113:0/3000929014 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f258006b350 msgr2=0x7f258006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:00.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.499+0000 7f25a4a38700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f258006b350 0x7f258006d810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f25a009b260 tx=0x7f2594008030 comp rx=0 tx=0).stop 2024-01-15T11:26:00.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.499+0000 7f25a4a38700 1 -- 172.21.15.113:0/3000929014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a00d22f0 msgr2=0x7f25a00ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:00.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.499+0000 7f25a4a38700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a00d22f0 0x7f25a00ca740 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f259000b4a0 tx=0x7f259000b860 comp rx=0 tx=0).stop 2024-01-15T11:26:00.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.499+0000 7f25a4a38700 1 -- 172.21.15.113:0/3000929014 shutdown_connections 2024-01-15T11:26:00.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.499+0000 7f25a4a38700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f258006b350 0x7f258006d810 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.499+0000 7f25a4a38700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25a015e080 0x7f25a015c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.499+0000 7f25a4a38700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25a00ce4d0 0x7f25a00b6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.499+0000 7f25a4a38700 1 --2- 172.21.15.113:0/3000929014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a00d22f0 0x7f25a00ca740 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.499+0000 7f25a4a38700 1 -- 172.21.15.113:0/3000929014 >> 172.21.15.113:0/3000929014 conn(0x7f25a0073130 msgr2=0x7f25a015eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:00.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.499+0000 7f25a4a38700 1 -- 172.21.15.113:0/3000929014 shutdown_connections 2024-01-15T11:26:00.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.499+0000 7f25a4a38700 1 -- 172.21.15.113:0/3000929014 wait complete. 2024-01-15T11:26:00.499 INFO:teuthology.orchestra.run.smithi113.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-15T11:26:00.511 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-15T11:26:00.539 INFO:tasks.ceph.mds.d.smithi141.stdout:starting mds.d at 2024-01-15T11:26:00.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.635+0000 7f54206eb700 1 Processor -- start 2024-01-15T11:26:00.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.635+0000 7f54206eb700 1 -- start start 2024-01-15T11:26:00.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f54206eb700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5418161690 0x7f54180d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f54206eb700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f5418161a70 0x7f54180d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f54206eb700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54180d1730 0x7f541809aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f54206eb700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54180d0a70 con 0x7f5418161690 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f54206eb700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54180d1c10 con 0x7f5418161a70 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f54206eb700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54180b2ab0 con 0x7f54180d1730 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541e487700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5418161690 0x7f54180d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541e487700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5418161690 0x7f54180d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33512/0 (socket says 172.21.15.113:33512) 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541e487700 1 -- 172.21.15.113:0/3820525613 learned_addr learned my addr 172.21.15.113:0/3820525613 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541e487700 1 -- 172.21.15.113:0/3820525613 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54180d1730 msgr2=0x7f541809aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541ec88700 1 --2- 172.21.15.113:0/3820525613 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54180d1730 0x7f541809aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541dc86700 1 --2- 172.21.15.113:0/3820525613 >> v2:172.21.15.141:3300/0 conn(0x7f5418161a70 0x7f54180d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541e487700 1 --2- 172.21.15.113:0/3820525613 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54180d1730 0x7f541809aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541e487700 1 -- 172.21.15.113:0/3820525613 >> v2:172.21.15.141:3300/0 conn(0x7f5418161a70 msgr2=0x7f54180d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541e487700 1 --2- 172.21.15.113:0/3820525613 >> v2:172.21.15.141:3300/0 conn(0x7f5418161a70 0x7f54180d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541e487700 1 -- 172.21.15.113:0/3820525613 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54180b2d90 con 0x7f5418161690 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541ec88700 1 --2- 172.21.15.113:0/3820525613 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54180d1730 0x7f541809aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541dc86700 1 --2- 172.21.15.113:0/3820525613 >> v2:172.21.15.141:3300/0 conn(0x7f5418161a70 0x7f54180d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541e487700 1 --2- 172.21.15.113:0/3820525613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5418161690 0x7f54180d0f20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5418072ec0 tx=0x7f5414009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5edf527c33c4f567 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.636+0000 7f541d485700 1 -- 172.21.15.113:0/3820525613 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541400b870 con 0x7f5418161690 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.637+0000 7f541d485700 1 -- 172.21.15.113:0/3820525613 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541400b9d0 con 0x7f5418161690 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.637+0000 7f541d485700 1 -- 172.21.15.113:0/3820525613 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541400bd60 con 0x7f5418161690 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.637+0000 7f54206eb700 1 -- 172.21.15.113:0/3820525613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5418161690 msgr2=0x7f54180d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.637+0000 7f54206eb700 1 --2- 172.21.15.113:0/3820525613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5418161690 0x7f54180d0f20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5418072ec0 tx=0x7f5414009bd0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.637+0000 7f54206eb700 1 -- 172.21.15.113:0/3820525613 shutdown_connections 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.637+0000 7f54206eb700 1 --2- 172.21.15.113:0/3820525613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5418161690 0x7f54180d0f20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.637+0000 7f54206eb700 1 --2- 172.21.15.113:0/3820525613 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f54180d1730 0x7f541809aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.637+0000 7f54206eb700 1 --2- 172.21.15.113:0/3820525613 >> v2:172.21.15.141:3300/0 conn(0x7f5418161a70 0x7f54180d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.637+0000 7f54206eb700 1 -- 172.21.15.113:0/3820525613 >> 172.21.15.113:0/3820525613 conn(0x7f54180d1e50 msgr2=0x7f54180d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.637+0000 7f54206eb700 1 -- 172.21.15.113:0/3820525613 shutdown_connections 2024-01-15T11:26:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.637+0000 7f54206eb700 1 -- 172.21.15.113:0/3820525613 wait complete. 2024-01-15T11:26:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.638+0000 7f54206eb700 1 Processor -- start 2024-01-15T11:26:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.638+0000 7f54206eb700 1 -- start start 2024-01-15T11:26:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f54206eb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54180d1730 0x7f5418153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f54206eb700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5418161690 0x7f54180b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f54206eb700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5418161a70 0x7f541815cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f54206eb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54180b53d0 con 0x7f54180d1730 2024-01-15T11:26:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f54206eb700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54180b56b0 con 0x7f5418161690 2024-01-15T11:26:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f54206eb700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54180b5990 con 0x7f5418161a70 2024-01-15T11:26:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f541e487700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54180d1730 0x7f5418153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f541e487700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54180d1730 0x7f5418153720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33516/0 (socket says 172.21.15.113:33516) 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f541e487700 1 -- 172.21.15.113:0/2644862118 learned_addr learned my addr 172.21.15.113:0/2644862118 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f541ec88700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5418161a70 0x7f541815cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f541dc86700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5418161690 0x7f54180b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f541e487700 1 -- 172.21.15.113:0/2644862118 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5418161a70 msgr2=0x7f541815cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f541e487700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5418161a70 0x7f541815cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f541e487700 1 -- 172.21.15.113:0/2644862118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5418161690 msgr2=0x7f54180b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.639+0000 7f541e487700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5418161690 0x7f54180b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.640+0000 7f541e487700 1 -- 172.21.15.113:0/2644862118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5414009550 con 0x7f54180d1730 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.640+0000 7f541e487700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54180d1730 0x7f5418153720 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5418072870 tx=0x7f5414004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.640+0000 7f540effd700 1 -- 172.21.15.113:0/2644862118 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414004310 con 0x7f54180d1730 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.640+0000 7f540effd700 1 -- 172.21.15.113:0/2644862118 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541401f070 con 0x7f54180d1730 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.640+0000 7f540effd700 1 -- 172.21.15.113:0/2644862118 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54140036a0 con 0x7f54180d1730 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.640+0000 7f54206eb700 1 -- 172.21.15.113:0/2644862118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54180c6840 con 0x7f54180d1730 2024-01-15T11:26:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.640+0000 7f54206eb700 1 -- 172.21.15.113:0/2644862118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54180b5030 con 0x7f54180d1730 2024-01-15T11:26:00.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.641+0000 7f54206eb700 1 -- 172.21.15.113:0/2644862118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54180c66a0 con 0x7f54180d1730 2024-01-15T11:26:00.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.641+0000 7f540effd700 1 -- 172.21.15.113:0/2644862118 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5414021030 con 0x7f54180d1730 2024-01-15T11:26:00.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.642+0000 7f540effd700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f53fc06b350 0x7f53fc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:00.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.642+0000 7f540effd700 1 -- 172.21.15.113:0/2644862118 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f541408b0b0 con 0x7f54180d1730 2024-01-15T11:26:00.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.642+0000 7f541dc86700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f53fc06b350 0x7f53fc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:00.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.642+0000 7f541dc86700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f53fc06b350 0x7f53fc06d810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5408006fd0 tx=0x7f5408008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:00.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.646+0000 7f540effd700 1 -- 172.21.15.113:0/2644862118 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54180c66a0 con 0x7f54180d1730 2024-01-15T11:26:00.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:00.878+0000 7f54206eb700 1 -- 172.21.15.113:0/2644862118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f54180c6a80 con 0x7f54180d1730 2024-01-15T11:26:01.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.498+0000 7f540effd700 1 -- 172.21.15.113:0/2644862118 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f54180c6a80 con 0x7f54180d1730 2024-01-15T11:26:01.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.501+0000 7f54206eb700 1 -- 172.21.15.113:0/2644862118 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f53fc06b350 msgr2=0x7f53fc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:01.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.501+0000 7f54206eb700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f53fc06b350 0x7f53fc06d810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5408006fd0 tx=0x7f5408008040 comp rx=0 tx=0).stop 2024-01-15T11:26:01.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.501+0000 7f54206eb700 1 -- 172.21.15.113:0/2644862118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54180d1730 msgr2=0x7f5418153720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:01.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.501+0000 7f54206eb700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54180d1730 0x7f5418153720 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5418072870 tx=0x7f5414004750 comp rx=0 tx=0).stop 2024-01-15T11:26:01.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.501+0000 7f54206eb700 1 -- 172.21.15.113:0/2644862118 shutdown_connections 2024-01-15T11:26:01.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.501+0000 7f54206eb700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f53fc06b350 0x7f53fc06d810 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:01.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.501+0000 7f54206eb700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f5418161a70 0x7f541815cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:01.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.501+0000 7f54206eb700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54180d1730 0x7f5418153720 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:01.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.501+0000 7f54206eb700 1 --2- 172.21.15.113:0/2644862118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5418161690 0x7f54180b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:01.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.501+0000 7f54206eb700 1 -- 172.21.15.113:0/2644862118 >> 172.21.15.113:0/2644862118 conn(0x7f54180d1e50 msgr2=0x7f541815e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:01.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.501+0000 7f54206eb700 1 -- 172.21.15.113:0/2644862118 shutdown_connections 2024-01-15T11:26:01.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.501+0000 7f54206eb700 1 -- 172.21.15.113:0/2644862118 wait complete. 2024-01-15T11:26:01.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-15T11:26:01.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.648+0000 7fa04ba9c700 1 Processor -- start 2024-01-15T11:26:01.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.649+0000 7fa04ba9c700 1 -- start start 2024-01-15T11:26:01.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.649+0000 7fa04ba9c700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa04415df20 0x7fa04406c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:01.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.649+0000 7fa04ba9c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa0441622b0 0x7fa04406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:01.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.649+0000 7fa04ba9c700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04406bed0 con 0x7fa044162820 2024-01-15T11:26:01.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.649+0000 7fa04ba9c700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa04406cfe0 con 0x7fa04415df20 2024-01-15T11:26:01.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.649+0000 7fa04ba9c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04406bba0 con 0x7fa0441622b0 2024-01-15T11:26:01.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa049037700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fa044162820 0x7fa044161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48374/0 (socket says 172.21.15.113:48374) 2024-01-15T11:26:01.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa049037700 1 -- 172.21.15.113:0/4264577525 learned_addr learned my addr 172.21.15.113:0/4264577525 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:01.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045152444 0 0) 0x7fa04406bed0 con 0x7fa044162820 2024-01-15T11:26:01.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0300035b0 con 0x7fa044162820 2024-01-15T11:26:01.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa04a039700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa0441622b0 0x7fa04406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:01.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa049838700 1 --2- 172.21.15.113:0/4264577525 >> v2:172.21.15.141:3300/0 conn(0x7fa04415df20 0x7fa04406c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:01.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2599028672 0 0) 0x7fa0300035b0 con 0x7fa044162820 2024-01-15T11:26:01.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa0441622b0 msgr2=0x7fa04406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:01.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa0441622b0 0x7fa04406caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:01.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 >> v2:172.21.15.141:3300/0 conn(0x7fa04415df20 msgr2=0x7fa04406c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:01.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 --2- 172.21.15.113:0/4264577525 >> v2:172.21.15.141:3300/0 conn(0x7fa04415df20 0x7fa04406c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:01.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa044070060 con 0x7fa044162820 2024-01-15T11:26:01.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fa040001670 con 0x7fa044162820 2024-01-15T11:26:01.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 >> v1:172.21.15.113:6789/0 conn(0x7fa044162820 legacy=0x7fa044161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:01.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa0441622b0 0x7fa030003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:01.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa030003f80 0x7fa030004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:01.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa030004d90 0x7fa030005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:01.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04406cfe0 con 0x7fa030003f80 2024-01-15T11:26:01.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04406bba0 con 0x7fa030004d90 2024-01-15T11:26:01.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03000ba60 con 0x7fa0441622b0 2024-01-15T11:26:01.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa049037700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa030003f80 0x7fa030004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:01.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa049838700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa0441622b0 0x7fa030003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:01.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa04a039700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa030004d90 0x7fa030005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:01.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa049037700 1 -- 172.21.15.113:0/4264577525 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa0441622b0 msgr2=0x7fa030003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:01.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa049037700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa0441622b0 0x7fa030003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:01.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.650+0000 7fa049037700 1 -- 172.21.15.113:0/4264577525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa030004d90 msgr2=0x7fa030005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:01.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa049037700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa030004d90 0x7fa030005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:01.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa049037700 1 -- 172.21.15.113:0/4264577525 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0300035b0 con 0x7fa030003f80 2024-01-15T11:26:01.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa04a039700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa030004d90 0x7fa030005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:01.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa049037700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa030003f80 0x7fa030004850 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa044162c60 tx=0x7fa030005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c332c94769b9110 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:01.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04000ce00 con 0x7fa030003f80 2024-01-15T11:26:01.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa040005960 con 0x7fa030003f80 2024-01-15T11:26:01.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa048836700 1 -- 172.21.15.113:0/4264577525 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040005d10 con 0x7fa030003f80 2024-01-15T11:26:01.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa04ba9c700 1 -- 172.21.15.113:0/4264577525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa030003f80 msgr2=0x7fa030004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:01.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa04ba9c700 1 --2- 172.21.15.113:0/4264577525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa030003f80 0x7fa030004850 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa044162c60 tx=0x7fa030005880 comp rx=0 tx=0).stop 2024-01-15T11:26:01.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa04a039700 1 -- 172.21.15.113:0/4264577525 reap_dead start 2024-01-15T11:26:01.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa04ba9c700 1 -- 172.21.15.113:0/4264577525 shutdown_connections 2024-01-15T11:26:01.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa04ba9c700 1 -- 172.21.15.113:0/4264577525 >> 172.21.15.113:0/4264577525 conn(0x7fa04406dc80 msgr2=0x7fa04409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:01.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa04ba9c700 1 -- 172.21.15.113:0/4264577525 shutdown_connections 2024-01-15T11:26:01.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.651+0000 7fa04ba9c700 1 -- 172.21.15.113:0/4264577525 wait complete. 2024-01-15T11:26:01.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.653+0000 7fa04ba9c700 1 Processor -- start 2024-01-15T11:26:01.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.653+0000 7fa04ba9c700 1 -- start start 2024-01-15T11:26:01.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa04ba9c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa030003f80 0x7fa044149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:01.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa04ba9c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0441ea690 0x7fa0441e8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:01.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa04ba9c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa04415cad0 0x7fa0440b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:01.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa04ba9c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0441ea200 con 0x7fa030003f80 2024-01-15T11:26:01.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa04ba9c700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa044149760 con 0x7fa0441ea690 2024-01-15T11:26:01.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa04ba9c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0441e95f0 con 0x7fa04415cad0 2024-01-15T11:26:01.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa049838700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa030003f80 0x7fa044149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:01.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa049838700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa030003f80 0x7fa044149220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33524/0 (socket says 172.21.15.113:33524) 2024-01-15T11:26:01.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa049838700 1 -- 172.21.15.113:0/1007690670 learned_addr learned my addr 172.21.15.113:0/1007690670 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:01.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa049838700 1 -- 172.21.15.113:0/1007690670 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa04415cad0 msgr2=0x7fa0440b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:01.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa049838700 1 --2- 172.21.15.113:0/1007690670 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa04415cad0 0x7fa0440b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:01.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa049838700 1 -- 172.21.15.113:0/1007690670 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0441ea690 msgr2=0x7fa0441e8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:01.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa049838700 1 --2- 172.21.15.113:0/1007690670 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0441ea690 0x7fa0441e8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:01.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa049838700 1 -- 172.21.15.113:0/1007690670 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0441e98d0 con 0x7fa030003f80 2024-01-15T11:26:01.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.654+0000 7fa049838700 1 --2- 172.21.15.113:0/1007690670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa030003f80 0x7fa044149220 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa034011880 tx=0x7fa034011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:01.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.655+0000 7fa03a7fc700 1 -- 172.21.15.113:0/1007690670 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03400c070 con 0x7fa030003f80 2024-01-15T11:26:01.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.655+0000 7fa03a7fc700 1 -- 172.21.15.113:0/1007690670 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03400f8d0 con 0x7fa030003f80 2024-01-15T11:26:01.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.655+0000 7fa04ba9c700 1 -- 172.21.15.113:0/1007690670 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0441e9bb0 con 0x7fa030003f80 2024-01-15T11:26:01.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.655+0000 7fa03a7fc700 1 -- 172.21.15.113:0/1007690670 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03400fc80 con 0x7fa030003f80 2024-01-15T11:26:01.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.655+0000 7fa04ba9c700 1 -- 172.21.15.113:0/1007690670 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0441e9270 con 0x7fa030003f80 2024-01-15T11:26:01.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.656+0000 7fa04ba9c700 1 -- 172.21.15.113:0/1007690670 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0441ea4e0 con 0x7fa030003f80 2024-01-15T11:26:01.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.656+0000 7fa03a7fc700 1 -- 172.21.15.113:0/1007690670 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa034015460 con 0x7fa030003f80 2024-01-15T11:26:01.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.656+0000 7fa03a7fc700 1 --2- 172.21.15.113:0/1007690670 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa02806b350 0x7fa02806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:01.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.656+0000 7fa03a7fc700 1 -- 172.21.15.113:0/1007690670 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fa034089f10 con 0x7fa030003f80 2024-01-15T11:26:01.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.656+0000 7fa049037700 1 --2- 172.21.15.113:0/1007690670 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa02806b350 0x7fa02806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:01.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.657+0000 7fa049037700 1 --2- 172.21.15.113:0/1007690670 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa02806b350 0x7fa02806d810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa040000c00 tx=0x7fa04000f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:01.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.661+0000 7fa03a7fc700 1 -- 172.21.15.113:0/1007690670 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa034054cb0 con 0x7fa030003f80 2024-01-15T11:26:01.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:01.851+0000 7fa04ba9c700 1 -- 172.21.15.113:0/1007690670 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa044067f60 con 0x7fa030003f80 2024-01-15T11:26:02.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.504+0000 7fa03a7fc700 1 -- 172.21.15.113:0/1007690670 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa034023020 con 0x7fa030003f80 2024-01-15T11:26:02.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.508+0000 7fa04ba9c700 1 -- 172.21.15.113:0/1007690670 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa02806b350 msgr2=0x7fa02806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:02.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.508+0000 7fa04ba9c700 1 --2- 172.21.15.113:0/1007690670 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa02806b350 0x7fa02806d810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa040000c00 tx=0x7fa04000f000 comp rx=0 tx=0).stop 2024-01-15T11:26:02.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.508+0000 7fa04ba9c700 1 -- 172.21.15.113:0/1007690670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa030003f80 msgr2=0x7fa044149220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:02.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.508+0000 7fa04ba9c700 1 --2- 172.21.15.113:0/1007690670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa030003f80 0x7fa044149220 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa034011880 tx=0x7fa034011c40 comp rx=0 tx=0).stop 2024-01-15T11:26:02.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.508+0000 7fa04ba9c700 1 -- 172.21.15.113:0/1007690670 shutdown_connections 2024-01-15T11:26:02.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.508+0000 7fa04ba9c700 1 --2- 172.21.15.113:0/1007690670 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa02806b350 0x7fa02806d810 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:02.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.508+0000 7fa04ba9c700 1 --2- 172.21.15.113:0/1007690670 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa04415cad0 0x7fa0440b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:02.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.508+0000 7fa04ba9c700 1 --2- 172.21.15.113:0/1007690670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa030003f80 0x7fa044149220 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:02.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.508+0000 7fa04ba9c700 1 --2- 172.21.15.113:0/1007690670 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa0441ea690 0x7fa0441e8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:02.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.508+0000 7fa04ba9c700 1 -- 172.21.15.113:0/1007690670 >> 172.21.15.113:0/1007690670 conn(0x7fa04406dc80 msgr2=0x7fa0440b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:02.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.508+0000 7fa04ba9c700 1 -- 172.21.15.113:0/1007690670 shutdown_connections 2024-01-15T11:26:02.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.508+0000 7fa04ba9c700 1 -- 172.21.15.113:0/1007690670 wait complete. 2024-01-15T11:26:02.509 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'cephfs_metadata' removed 2024-01-15T11:26:02.523 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-15T11:26:02.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.652+0000 7fc3f1892700 1 Processor -- start 2024-01-15T11:26:02.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.652+0000 7fc3f1892700 1 -- start start 2024-01-15T11:26:02.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.653+0000 7fc3f1892700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc3ec15df20 0x7fc3ec06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:02.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.653+0000 7fc3f1892700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1622b0 0x7fc3ec06caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:02.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.653+0000 7fc3f1892700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ec06bed0 con 0x7fc3ec162820 2024-01-15T11:26:02.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.653+0000 7fc3f1892700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ec06cfe0 con 0x7fc3ec15df20 2024-01-15T11:26:02.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.653+0000 7fc3f1892700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec06bba0 con 0x7fc3ec1622b0 2024-01-15T11:26:02.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.653+0000 7fc3ea7fc700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fc3ec162820 0x7fc3ec161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48386/0 (socket says 172.21.15.113:48386) 2024-01-15T11:26:02.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.653+0000 7fc3ea7fc700 1 -- 172.21.15.113:0/3340629055 learned_addr learned my addr 172.21.15.113:0/3340629055 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:02.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.653+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692214646 0 0) 0x7fc3ec06bed0 con 0x7fc3ec162820 2024-01-15T11:26:02.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d40035b0 con 0x7fc3ec162820 2024-01-15T11:26:02.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3eb7fe700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1622b0 0x7fc3ec06caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:02.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3eaffd700 1 --2- 172.21.15.113:0/3340629055 >> v2:172.21.15.141:3300/0 conn(0x7fc3ec15df20 0x7fc3ec06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:02.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 313096211 0 0) 0x7fc3d40035b0 con 0x7fc3ec162820 2024-01-15T11:26:02.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1622b0 msgr2=0x7fc3ec06caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:02.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1622b0 0x7fc3ec06caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:02.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 >> v2:172.21.15.141:3300/0 conn(0x7fc3ec15df20 msgr2=0x7fc3ec06c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:02.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 --2- 172.21.15.113:0/3340629055 >> v2:172.21.15.141:3300/0 conn(0x7fc3ec15df20 0x7fc3ec06c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:02.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec070060 con 0x7fc3ec162820 2024-01-15T11:26:02.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fc3dc001670 con 0x7fc3ec162820 2024-01-15T11:26:02.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3eb7fe700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1622b0 0x7fc3ec06caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:02.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3eaffd700 1 --2- 172.21.15.113:0/3340629055 >> v2:172.21.15.141:3300/0 conn(0x7fc3ec15df20 0x7fc3ec06c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:02.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 >> v1:172.21.15.113:6789/0 conn(0x7fc3ec162820 legacy=0x7fc3ec161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3d4003a00 0x7fc3d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d4004850 0x7fc3d4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1622b0 0x7fc3d4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec06cfe0 con 0x7fc3d4003a00 2024-01-15T11:26:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec06bba0 con 0x7fc3d4004850 2024-01-15T11:26:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d400bac0 con 0x7fc3ec1622b0 2024-01-15T11:26:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3eaffd700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3d4003a00 0x7fc3d4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3eb7fe700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1622b0 0x7fc3d4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3ea7fc700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d4004850 0x7fc3d4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3eaffd700 1 -- 172.21.15.113:0/3340629055 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1622b0 msgr2=0x7fc3d4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3eaffd700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1622b0 0x7fc3d4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3eaffd700 1 -- 172.21.15.113:0/3340629055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d4004850 msgr2=0x7fc3d4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3eaffd700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d4004850 0x7fc3d4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3eaffd700 1 -- 172.21.15.113:0/3340629055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d40035b0 con 0x7fc3d4003a00 2024-01-15T11:26:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3eaffd700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3d4003a00 0x7fc3d4003e20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009950 tx=0x7fc3d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0692f86d1bc942b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.654+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e000c070 con 0x7fc3d4003a00 2024-01-15T11:26:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.655+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e0013610 con 0x7fc3d4003a00 2024-01-15T11:26:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.655+0000 7fc3e9ffb700 1 -- 172.21.15.113:0/3340629055 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0013980 con 0x7fc3d4003a00 2024-01-15T11:26:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.655+0000 7fc3f1892700 1 -- 172.21.15.113:0/3340629055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3d4003a00 msgr2=0x7fc3d4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.655+0000 7fc3f1892700 1 --2- 172.21.15.113:0/3340629055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3d4003a00 0x7fc3d4003e20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009950 tx=0x7fc3d4005880 comp rx=0 tx=0).stop 2024-01-15T11:26:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.655+0000 7fc3eb7fe700 1 -- 172.21.15.113:0/3340629055 reap_dead start 2024-01-15T11:26:02.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.655+0000 7fc3f1892700 1 -- 172.21.15.113:0/3340629055 shutdown_connections 2024-01-15T11:26:02.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.655+0000 7fc3f1892700 1 -- 172.21.15.113:0/3340629055 >> 172.21.15.113:0/3340629055 conn(0x7fc3ec06dc80 msgr2=0x7fc3ec09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:02.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.655+0000 7fc3f1892700 1 -- 172.21.15.113:0/3340629055 shutdown_connections 2024-01-15T11:26:02.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.655+0000 7fc3f1892700 1 -- 172.21.15.113:0/3340629055 wait complete. 2024-01-15T11:26:02.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.657+0000 7fc3f1892700 1 Processor -- start 2024-01-15T11:26:02.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.657+0000 7fc3f1892700 1 -- start start 2024-01-15T11:26:02.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3f1892700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d4003a00 0x7fc3ec1f46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:02.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3f1892700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1f1080 0x7fc3ec1f7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:02.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3f1892700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3ec1f77a0 0x7fc3ec0d1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:02.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3f1892700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec1f4c30 con 0x7fc3ec1f77a0 2024-01-15T11:26:02.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3f1892700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec1f3c70 con 0x7fc3d4003a00 2024-01-15T11:26:02.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3f1892700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec1f7d20 con 0x7fc3ec1f1080 2024-01-15T11:26:02.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3eb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3ec1f77a0 0x7fc3ec0d1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:02.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3eb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3ec1f77a0 0x7fc3ec0d1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33538/0 (socket says 172.21.15.113:33538) 2024-01-15T11:26:02.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3eb7fe700 1 -- 172.21.15.113:0/318100574 learned_addr learned my addr 172.21.15.113:0/318100574 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:02.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3ea7fc700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1f1080 0x7fc3ec1f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:02.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3eaffd700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d4003a00 0x7fc3ec1f46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:02.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3eb7fe700 1 -- 172.21.15.113:0/318100574 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1f1080 msgr2=0x7fc3ec1f7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:02.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3eb7fe700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1f1080 0x7fc3ec1f7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:02.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3eb7fe700 1 -- 172.21.15.113:0/318100574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d4003a00 msgr2=0x7fc3ec1f46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:02.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3eb7fe700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d4003a00 0x7fc3ec1f46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:02.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3eb7fe700 1 -- 172.21.15.113:0/318100574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec1f8000 con 0x7fc3ec1f77a0 2024-01-15T11:26:02.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3ea7fc700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1f1080 0x7fc3ec1f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:02.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3eaffd700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d4003a00 0x7fc3ec1f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:02.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3eb7fe700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3ec1f77a0 0x7fc3ec0d1200 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec09ff50 tx=0x7fc3e4010440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:02.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3d3fff700 1 -- 172.21.15.113:0/318100574 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e401b070 con 0x7fc3ec1f77a0 2024-01-15T11:26:02.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3d3fff700 1 -- 172.21.15.113:0/318100574 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e4010b30 con 0x7fc3ec1f77a0 2024-01-15T11:26:02.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3d3fff700 1 -- 172.21.15.113:0/318100574 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4010ce0 con 0x7fc3ec1f77a0 2024-01-15T11:26:02.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3f1892700 1 -- 172.21.15.113:0/318100574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ec1f3ef0 con 0x7fc3ec1f77a0 2024-01-15T11:26:02.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3f1892700 1 -- 172.21.15.113:0/318100574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ec15b590 con 0x7fc3ec1f77a0 2024-01-15T11:26:02.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.659+0000 7fc3f1892700 1 -- 172.21.15.113:0/318100574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ec1f81f0 con 0x7fc3ec1f77a0 2024-01-15T11:26:02.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.660+0000 7fc3d3fff700 1 -- 172.21.15.113:0/318100574 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc3e4017430 con 0x7fc3ec1f77a0 2024-01-15T11:26:02.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.660+0000 7fc3d3fff700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc3d806b350 0x7fc3d806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:02.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.660+0000 7fc3d3fff700 1 -- 172.21.15.113:0/318100574 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5473+0+0 (secure 0 0 0) 0x7fc3e4089e70 con 0x7fc3ec1f77a0 2024-01-15T11:26:02.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.660+0000 7fc3eaffd700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc3d806b350 0x7fc3d806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:02.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.661+0000 7fc3eaffd700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc3d806b350 0x7fc3d806d810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0000c00 tx=0x7fc3e000c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:02.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.665+0000 7fc3d3fff700 1 -- 172.21.15.113:0/318100574 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3ec1f81f0 con 0x7fc3ec1f77a0 2024-01-15T11:26:02.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:02.861+0000 7fc3f1892700 1 -- 172.21.15.113:0/318100574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc3ec1f81f0 con 0x7fc3ec1f77a0 2024-01-15T11:26:03.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.507+0000 7fc3d3fff700 1 -- 172.21.15.113:0/318100574 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc3ec1f81f0 con 0x7fc3ec1f77a0 2024-01-15T11:26:03.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.510+0000 7fc3f1892700 1 -- 172.21.15.113:0/318100574 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc3d806b350 msgr2=0x7fc3d806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:03.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.510+0000 7fc3f1892700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc3d806b350 0x7fc3d806d810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0000c00 tx=0x7fc3e000c000 comp rx=0 tx=0).stop 2024-01-15T11:26:03.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.510+0000 7fc3f1892700 1 -- 172.21.15.113:0/318100574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3ec1f77a0 msgr2=0x7fc3ec0d1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:03.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.510+0000 7fc3f1892700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3ec1f77a0 0x7fc3ec0d1200 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec09ff50 tx=0x7fc3e4010440 comp rx=0 tx=0).stop 2024-01-15T11:26:03.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.510+0000 7fc3f1892700 1 -- 172.21.15.113:0/318100574 shutdown_connections 2024-01-15T11:26:03.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.510+0000 7fc3f1892700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc3d806b350 0x7fc3d806d810 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:03.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.510+0000 7fc3f1892700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3ec1f77a0 0x7fc3ec0d1200 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:03.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.510+0000 7fc3f1892700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc3d4003a00 0x7fc3ec1f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:03.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.510+0000 7fc3f1892700 1 --2- 172.21.15.113:0/318100574 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc3ec1f1080 0x7fc3ec1f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:03.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.510+0000 7fc3f1892700 1 -- 172.21.15.113:0/318100574 >> 172.21.15.113:0/318100574 conn(0x7fc3ec06dc80 msgr2=0x7fc3ec15c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:03.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.510+0000 7fc3f1892700 1 -- 172.21.15.113:0/318100574 shutdown_connections 2024-01-15T11:26:03.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.510+0000 7fc3f1892700 1 -- 172.21.15.113:0/318100574 wait complete. 2024-01-15T11:26:03.510 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'cephfs_data' removed 2024-01-15T11:26:03.524 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T11:26:03.524 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T11:26:03.525 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T11:26:03.525 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T11:26:03.525 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T11:26:03.525 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T11:26:03.525 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T11:26:03.526 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T11:26:03.526 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T11:26:03.526 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T11:26:03.526 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T11:26:03.526 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T11:26:03.527 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T11:26:03.527 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T11:26:03.527 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T11:26:03.527 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T11:26:03.536 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T11:26:03.536 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T11:26:03.537 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T11:26:03.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T11:26:03.541 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T11:26:03.541 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T11:26:03.542 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T11:26:03.542 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T11:26:03.578 INFO:tasks.ceph.mds.a:Started 2024-01-15T11:26:03.579 DEBUG:teuthology.parallel:result is None 2024-01-15T11:26:03.579 INFO:tasks.ceph.mds.c:Started 2024-01-15T11:26:03.580 DEBUG:teuthology.parallel:result is None 2024-01-15T11:26:03.583 INFO:tasks.ceph.mds.b:Started 2024-01-15T11:26:03.584 DEBUG:teuthology.parallel:result is None 2024-01-15T11:26:03.584 INFO:tasks.ceph.mds.d:Started 2024-01-15T11:26:03.585 DEBUG:teuthology.parallel:result is None 2024-01-15T11:26:03.585 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T11:26:03.664 INFO:tasks.ceph.mds.c.smithi113.stdout:starting mds.c at 2024-01-15T11:26:03.664 INFO:tasks.ceph.mds.a.smithi113.stdout:starting mds.a at 2024-01-15T11:26:03.669 INFO:tasks.ceph.mds.b.smithi141.stdout:starting mds.b at 2024-01-15T11:26:03.669 INFO:tasks.ceph.mds.d.smithi141.stdout:starting mds.d at 2024-01-15T11:26:03.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.720+0000 7f8aa9ead700 1 Processor -- start 2024-01-15T11:26:03.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.720+0000 7f8aa9ead700 1 -- start start 2024-01-15T11:26:03.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa9ead700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f8aa415bd60 0x7f8aa40d1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:03.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa9ead700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa40cc780 con 0x7f8aa415c140 2024-01-15T11:26:03.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa9ead700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa40cc9b0 con 0x7f8aa40cf2b0 2024-01-15T11:26:03.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa9ead700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa40ccc90 con 0x7f8aa415bd60 2024-01-15T11:26:03.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa2ffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f8aa415c140 0x7f8aa40cda50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:48398/0 (socket says 172.21.15.113:48398) 2024-01-15T11:26:03.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa2ffd700 1 -- 172.21.15.113:0/3917004878 learned_addr learned my addr 172.21.15.113:0/3917004878 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:03.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa37fe700 1 --2- 172.21.15.113:0/3917004878 >> v2:172.21.15.141:3300/0 conn(0x7f8aa415bd60 0x7f8aa40d1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:03.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779443848 0 0) 0x7f8aa40cc780 con 0x7f8aa415c140 2024-01-15T11:26:03.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a8c0035b0 con 0x7f8aa415c140 2024-01-15T11:26:03.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2239975749 0 0) 0x7f8a8c0035b0 con 0x7f8aa415c140 2024-01-15T11:26:03.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 >> v2:172.21.15.141:3300/0 conn(0x7f8aa415bd60 msgr2=0x7f8aa40d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:03.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa27fc700 1 --2- 172.21.15.113:0/3917004878 >> v2:172.21.15.141:3300/0 conn(0x7f8aa415bd60 0x7f8aa40d1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:03.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 >> v1:172.21.15.141:6789/0 conn(0x7f8aa40cf2b0 legacy=0x7f8aa40c57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:03.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa40ccf70 con 0x7f8aa415c140 2024-01-15T11:26:03.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f8a98001670 con 0x7f8aa415c140 2024-01-15T11:26:03.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 >> v1:172.21.15.113:6789/0 conn(0x7f8aa415c140 legacy=0x7f8aa40cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:03.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa27fc700 1 --2- 172.21.15.113:0/3917004878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a8c003a00 0x7f8a8c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:03.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa27fc700 1 --2- 172.21.15.113:0/3917004878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 0x7f8a8c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:03.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.721+0000 7f8aa27fc700 1 --2- 172.21.15.113:0/3917004878 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8a8c005210 0x7f8a8c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:03.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa40ccc90 con 0x7f8a8c004850 2024-01-15T11:26:03.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c00c0b0 con 0x7f8a8c003a00 2024-01-15T11:26:03.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c00c270 con 0x7f8a8c005210 2024-01-15T11:26:03.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa2ffd700 1 --2- 172.21.15.113:0/3917004878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 0x7f8a8c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:03.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa37fe700 1 --2- 172.21.15.113:0/3917004878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a8c003a00 0x7f8a8c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:03.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa3fff700 1 --2- 172.21.15.113:0/3917004878 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8a8c005210 0x7f8a8c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:03.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa2ffd700 1 -- 172.21.15.113:0/3917004878 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8a8c005210 msgr2=0x7f8a8c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:03.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa2ffd700 1 --2- 172.21.15.113:0/3917004878 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8a8c005210 0x7f8a8c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:03.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa2ffd700 1 -- 172.21.15.113:0/3917004878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a8c003a00 msgr2=0x7f8a8c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:03.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa2ffd700 1 --2- 172.21.15.113:0/3917004878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8a8c003a00 0x7f8a8c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:03.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa2ffd700 1 -- 172.21.15.113:0/3917004878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8c0035b0 con 0x7f8a8c004850 2024-01-15T11:26:03.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa3fff700 1 -- 172.21.15.113:0/3917004878 reap_dead start 2024-01-15T11:26:03.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa2ffd700 1 --2- 172.21.15.113:0/3917004878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 0x7f8a8c004cd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800c240 tx=0x7f8a8c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd4248fc6085c1ef server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:03.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98007c10 con 0x7f8a8c004850 2024-01-15T11:26:03.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9800ce00 con 0x7f8a8c004850 2024-01-15T11:26:03.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.722+0000 7f8aa27fc700 1 -- 172.21.15.113:0/3917004878 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98005ba0 con 0x7f8a8c004850 2024-01-15T11:26:03.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.723+0000 7f8aa9ead700 1 -- 172.21.15.113:0/3917004878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 msgr2=0x7f8a8c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:03.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.723+0000 7f8aa9ead700 1 --2- 172.21.15.113:0/3917004878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 0x7f8a8c004cd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800c240 tx=0x7f8a8c005880 comp rx=0 tx=0).stop 2024-01-15T11:26:03.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.723+0000 7f8aa9ead700 1 -- 172.21.15.113:0/3917004878 shutdown_connections 2024-01-15T11:26:03.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.723+0000 7f8aa9ead700 1 --2- 172.21.15.113:0/3917004878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 0x7f8a8c004cd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:03.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.723+0000 7f8aa9ead700 1 -- 172.21.15.113:0/3917004878 >> 172.21.15.113:0/3917004878 conn(0x7f8aa4073130 msgr2=0x7f8aa409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:03.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.723+0000 7f8aa9ead700 1 -- 172.21.15.113:0/3917004878 shutdown_connections 2024-01-15T11:26:03.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.723+0000 7f8aa9ead700 1 -- 172.21.15.113:0/3917004878 wait complete. 2024-01-15T11:26:03.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.724+0000 7f8aa9ead700 1 Processor -- start 2024-01-15T11:26:03.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.724+0000 7f8aa9ead700 1 -- start start 2024-01-15T11:26:03.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa9ead700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 0x7f8aa409f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:03.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa9ead700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa40b3c70 0x7f8aa41f2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:03.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa9ead700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8aa40b5a90 0x7f8aa40b5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:03.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa9ead700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa41f28a0 con 0x7f8a8c004850 2024-01-15T11:26:03.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa9ead700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa40b6450 con 0x7f8aa40b3c70 2024-01-15T11:26:03.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa9ead700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa40b4150 con 0x7f8aa40b5a90 2024-01-15T11:26:03.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 0x7f8aa409f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:03.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 0x7f8aa409f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33582/0 (socket says 172.21.15.113:33582) 2024-01-15T11:26:03.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa37fe700 1 -- 172.21.15.113:0/1173872526 learned_addr learned my addr 172.21.15.113:0/1173872526 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:03.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa2ffd700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa40b3c70 0x7f8aa41f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:03.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa37fe700 1 -- 172.21.15.113:0/1173872526 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8aa40b5a90 msgr2=0x7f8aa40b5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:03.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa37fe700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8aa40b5a90 0x7f8aa40b5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:03.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa37fe700 1 -- 172.21.15.113:0/1173872526 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa40b3c70 msgr2=0x7f8aa41f2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:03.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa37fe700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa40b3c70 0x7f8aa41f2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:03.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa37fe700 1 -- 172.21.15.113:0/1173872526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa40b4430 con 0x7f8a8c004850 2024-01-15T11:26:03.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa37fe700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 0x7f8aa409f6f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8a94011aa0 tx=0x7f8a94011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:03.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8a8bfff700 1 -- 172.21.15.113:0/1173872526 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400f9e0 con 0x7f8a8c004850 2024-01-15T11:26:03.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8a8bfff700 1 -- 172.21.15.113:0/1173872526 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9400fb40 con 0x7f8a8c004850 2024-01-15T11:26:03.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa9ead700 1 -- 172.21.15.113:0/1173872526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa40b4710 con 0x7f8a8c004850 2024-01-15T11:26:03.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8a8bfff700 1 -- 172.21.15.113:0/1173872526 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a94014440 con 0x7f8a8c004850 2024-01-15T11:26:03.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.725+0000 7f8aa9ead700 1 -- 172.21.15.113:0/1173872526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa40b4cd0 con 0x7f8a8c004850 2024-01-15T11:26:03.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.726+0000 7f8aa9ead700 1 -- 172.21.15.113:0/1173872526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa40c8d20 con 0x7f8a8c004850 2024-01-15T11:26:03.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.727+0000 7f8a8bfff700 1 -- 172.21.15.113:0/1173872526 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8a940145a0 con 0x7f8a8c004850 2024-01-15T11:26:03.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.727+0000 7f8a8bfff700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8a9006b350 0x7f8a9006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:03.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.727+0000 7f8a8bfff700 1 -- 172.21.15.113:0/1173872526 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8a9408a020 con 0x7f8a8c004850 2024-01-15T11:26:03.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.727+0000 7f8aa2ffd700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8a9006b350 0x7f8a9006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:03.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.728+0000 7f8aa2ffd700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8a9006b350 0x7f8a9006d810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8a98015040 tx=0x7f8a98007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:03.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.732+0000 7f8a8bfff700 1 -- 172.21.15.113:0/1173872526 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a94055220 con 0x7f8a8c004850 2024-01-15T11:26:03.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:03.937+0000 7f8aa9ead700 1 -- 172.21.15.113:0/1173872526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8aa4067f60 con 0x7f8a8c004850 2024-01-15T11:26:04.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.516+0000 7f8a8bfff700 1 -- 172.21.15.113:0/1173872526 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8a94058840 con 0x7f8a8c004850 2024-01-15T11:26:04.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.520+0000 7f8aa9ead700 1 -- 172.21.15.113:0/1173872526 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8a9006b350 msgr2=0x7f8a9006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:04.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.520+0000 7f8aa9ead700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8a9006b350 0x7f8a9006d810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8a98015040 tx=0x7f8a98007000 comp rx=0 tx=0).stop 2024-01-15T11:26:04.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.520+0000 7f8aa9ead700 1 -- 172.21.15.113:0/1173872526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 msgr2=0x7f8aa409f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:04.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.520+0000 7f8aa9ead700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 0x7f8aa409f6f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8a94011aa0 tx=0x7f8a94011e60 comp rx=0 tx=0).stop 2024-01-15T11:26:04.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.520+0000 7f8aa9ead700 1 -- 172.21.15.113:0/1173872526 shutdown_connections 2024-01-15T11:26:04.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.520+0000 7f8aa9ead700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f8a9006b350 0x7f8a9006d810 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.520+0000 7f8aa9ead700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f8aa40b5a90 0x7f8aa40b5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.520+0000 7f8aa9ead700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a8c004850 0x7f8aa409f6f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.520+0000 7f8aa9ead700 1 --2- 172.21.15.113:0/1173872526 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8aa40b3c70 0x7f8aa41f2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.520+0000 7f8aa9ead700 1 -- 172.21.15.113:0/1173872526 >> 172.21.15.113:0/1173872526 conn(0x7f8aa4073130 msgr2=0x7f8aa40c6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:04.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.520+0000 7f8aa9ead700 1 -- 172.21.15.113:0/1173872526 shutdown_connections 2024-01-15T11:26:04.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.520+0000 7f8aa9ead700 1 -- 172.21.15.113:0/1173872526 wait complete. 2024-01-15T11:26:04.519 INFO:teuthology.orchestra.run.smithi113.stderr: removed all blocklist entries 2024-01-15T11:26:04.531 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T11:26:04.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.654+0000 7fec18af2700 1 Processor -- start 2024-01-15T11:26:04.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.654+0000 7fec18af2700 1 -- start start 2024-01-15T11:26:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.655+0000 7fec18af2700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fec1415bed0 0x7fec140b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.655+0000 7fec18af2700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fec141603c0 0x7fec140d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.655+0000 7fec18af2700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec140d0770 con 0x7fec140d1a50 2024-01-15T11:26:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.655+0000 7fec18af2700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec1415c2b0 con 0x7fec141603c0 2024-01-15T11:26:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.655+0000 7fec18af2700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec140cd100 con 0x7fec1415bed0 2024-01-15T11:26:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.655+0000 7fec11d9b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fec141603c0 0x7fec140d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.655+0000 7fec12d9d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fec140d1a50 0x7fec140d1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:43418/0 (socket says 172.21.15.113:43418) 2024-01-15T11:26:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.655+0000 7fec12d9d700 1 -- 172.21.15.113:0/871865894 learned_addr learned my addr 172.21.15.113:0/871865894 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.656+0000 7fec1259c700 1 --2- 172.21.15.113:0/871865894 >> v2:172.21.15.141:3300/0 conn(0x7fec1415bed0 0x7fec140b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.656+0000 7fec11d9b700 1 -- 172.21.15.113:0/871865894 >> v2:172.21.15.141:3300/0 conn(0x7fec1415bed0 msgr2=0x7fec140b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.656+0000 7fec11d9b700 1 --2- 172.21.15.113:0/871865894 >> v2:172.21.15.141:3300/0 conn(0x7fec1415bed0 0x7fec140b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.656+0000 7fec11d9b700 1 -- 172.21.15.113:0/871865894 >> v1:172.21.15.113:6789/0 conn(0x7fec140d1a50 legacy=0x7fec140d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.656+0000 7fec1159a700 1 -- 172.21.15.113:0/871865894 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133580446 0 0) 0x7fec140d0770 con 0x7fec140d1a50 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.656+0000 7fec11d9b700 1 -- 172.21.15.113:0/871865894 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec140d09a0 con 0x7fec141603c0 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.656+0000 7fec11d9b700 1 --2- 172.21.15.113:0/871865894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec141603c0 0x7fec140d0230 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fec1409b2a0 tx=0x7fec04009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185846b9b5b5fa64 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.656+0000 7fec1159a700 1 -- 172.21.15.113:0/871865894 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec04011830 con 0x7fec141603c0 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.656+0000 7fec1159a700 1 -- 172.21.15.113:0/871865894 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec04011990 con 0x7fec141603c0 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.656+0000 7fec1159a700 1 -- 172.21.15.113:0/871865894 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec04011d20 con 0x7fec141603c0 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.657+0000 7fec18af2700 1 -- 172.21.15.113:0/871865894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec141603c0 msgr2=0x7fec140d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.657+0000 7fec18af2700 1 --2- 172.21.15.113:0/871865894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec141603c0 0x7fec140d0230 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fec1409b2a0 tx=0x7fec04009b20 comp rx=0 tx=0).stop 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.657+0000 7fec18af2700 1 -- 172.21.15.113:0/871865894 shutdown_connections 2024-01-15T11:26:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.657+0000 7fec18af2700 1 --2- 172.21.15.113:0/871865894 >> v2:172.21.15.141:3300/0 conn(0x7fec1415bed0 0x7fec140b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.657+0000 7fec18af2700 1 --2- 172.21.15.113:0/871865894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec141603c0 0x7fec140d0230 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.657+0000 7fec18af2700 1 -- 172.21.15.113:0/871865894 >> 172.21.15.113:0/871865894 conn(0x7fec14073130 msgr2=0x7fec1409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:04.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.657+0000 7fec18af2700 1 -- 172.21.15.113:0/871865894 shutdown_connections 2024-01-15T11:26:04.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.657+0000 7fec18af2700 1 -- 172.21.15.113:0/871865894 wait complete. 2024-01-15T11:26:04.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.658+0000 7fec18af2700 1 Processor -- start 2024-01-15T11:26:04.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.658+0000 7fec18af2700 1 -- start start 2024-01-15T11:26:04.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec18af2700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec140d1a50 0x7fec140af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec18af2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec1415bed0 0x7fec140af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec18af2700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fec141603c0 0x7fec1406bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec18af2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec140c5a80 con 0x7fec1415bed0 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec18af2700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1415cbd0 con 0x7fec140d1a50 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec18af2700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec140c4b70 con 0x7fec141603c0 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec11d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec1415bed0 0x7fec140af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec11d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec1415bed0 0x7fec140af9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43376/0 (socket says 172.21.15.113:43376) 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec11d9b700 1 -- 172.21.15.113:0/4131461118 learned_addr learned my addr 172.21.15.113:0/4131461118 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec11d9b700 1 -- 172.21.15.113:0/4131461118 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fec141603c0 msgr2=0x7fec1406bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec11d9b700 1 --2- 172.21.15.113:0/4131461118 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fec141603c0 0x7fec1406bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec11d9b700 1 -- 172.21.15.113:0/4131461118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec140d1a50 msgr2=0x7fec140af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec11d9b700 1 --2- 172.21.15.113:0/4131461118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec140d1a50 0x7fec140af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec11d9b700 1 -- 172.21.15.113:0/4131461118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec040094a0 con 0x7fec1415bed0 2024-01-15T11:26:04.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec11d9b700 1 --2- 172.21.15.113:0/4131461118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec1415bed0 0x7fec140af9d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fec04014040 tx=0x7fec04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:04.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec02ffd700 1 -- 172.21.15.113:0/4131461118 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec040038a0 con 0x7fec1415bed0 2024-01-15T11:26:04.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec02ffd700 1 -- 172.21.15.113:0/4131461118 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0401e070 con 0x7fec1415bed0 2024-01-15T11:26:04.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec02ffd700 1 -- 172.21.15.113:0/4131461118 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec04003c20 con 0x7fec1415bed0 2024-01-15T11:26:04.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec18af2700 1 -- 172.21.15.113:0/4131461118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec140aff10 con 0x7fec1415bed0 2024-01-15T11:26:04.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.659+0000 7fec18af2700 1 -- 172.21.15.113:0/4131461118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec140b01c0 con 0x7fec1415bed0 2024-01-15T11:26:04.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.660+0000 7fec18af2700 1 -- 172.21.15.113:0/4131461118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec14067f60 con 0x7fec1415bed0 2024-01-15T11:26:04.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.661+0000 7fec02ffd700 1 -- 172.21.15.113:0/4131461118 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fec0402d450 con 0x7fec1415bed0 2024-01-15T11:26:04.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.661+0000 7fec02ffd700 1 --2- 172.21.15.113:0/4131461118 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7febf406b350 0x7febf406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:04.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.661+0000 7fec02ffd700 1 -- 172.21.15.113:0/4131461118 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fec0408a8e0 con 0x7fec1415bed0 2024-01-15T11:26:04.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.661+0000 7fec1259c700 1 --2- 172.21.15.113:0/4131461118 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7febf406b350 0x7febf406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:04.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.662+0000 7fec1259c700 1 --2- 172.21.15.113:0/4131461118 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7febf406b350 0x7febf406d810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fec080097f0 tx=0x7fec08008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:04.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.666+0000 7fec02ffd700 1 -- 172.21.15.113:0/4131461118 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec04055a90 con 0x7fec1415bed0 2024-01-15T11:26:04.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.932+0000 7fec18af2700 1 -- 172.21.15.113:0/4131461118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fec1415cd10 con 0x7fec1415bed0 2024-01-15T11:26:04.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.932+0000 7fec02ffd700 1 -- 172.21.15.113:0/4131461118 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fec1415cd10 con 0x7fec1415bed0 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: "auth_dump": [ 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "mds.a", 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDgFaVl3qtfJBAApmUPtVUchbLyUR5i7tMBcQ==", 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow *", 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow *", 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.931 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.932 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "mds.b", 2024-01-15T11:26:04.932 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDgFaVll1DvMBAAyj6YKcqFWrYiuWCnEzfZyQ==", 2024-01-15T11:26:04.932 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.932 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-15T11:26:04.932 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow *", 2024-01-15T11:26:04.932 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow *", 2024-01-15T11:26:04.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.932 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.932 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.932 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.932 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "mds.c", 2024-01-15T11:26:04.932 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDgFaVlYja2KxAAVvtunMrSignFsgv9+m6zqw==", 2024-01-15T11:26:04.933 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.933 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-15T11:26:04.933 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow *", 2024-01-15T11:26:04.933 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow *", 2024-01-15T11:26:04.933 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.933 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.933 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.933 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.933 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "mds.d", 2024-01-15T11:26:04.933 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDgFaVl4jdhOBAAHlJN1TpsKN9lU8hjxWW89w==", 2024-01-15T11:26:04.933 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.933 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow *", 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow *", 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.0", 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDkFaVlVLNkHRAAsJe9+2Vra6oX5NYS69d1IQ==", 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.934 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.1", 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDnFaVlQ5v2GxAADh/Lh54BWAWEqoBa7khCZQ==", 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.2", 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDqFaVl2z7GGhAAwW5bYhZYJdr+UX0VaFseSg==", 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.935 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.3", 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDtFaVl6/ZmGhAAhk5GCH8FAsH8mGU/6BjG0w==", 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.936 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.4", 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDzFaVldKVuHhAAQV5AFYI6jisYR5QFMqpUUg==", 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.5", 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQD2FaVl7KklHRAA2VXFQIclhfkuliHmpWaK4g==", 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.937 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.6", 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQD5FaVl9m3NGxAAfiz7aeLWympZqVOfV19K4Q==", 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.7", 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQD8FaVlKbH8GhAAf+GPFNeLp2+s3jTtqklCmw==", 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.938 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.0", 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDhFaVl1ysgARAAgjFqY3kbPDbaQ0A6WxgMLQ==", 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow r", 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow rw", 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow rwx" 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.1", 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDhFaVlS0RDChAAfGRj4v0fLzIClr6FnNmemw==", 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow r", 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow rw", 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow rwx" 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.2", 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDhFaVlHvXmBhAAttmTrSddVurihd5SlwQhVw==", 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-15T11:26:04.940 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow r", 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow rw", 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow rwx" 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.3", 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDhFaVluyUHEBAA26iiKREXkJTEjQePzxY4JQ==", 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow r", 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow rw", 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow rwx" 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.941 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.admin", 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDgFaVlWI+jBhAAQMolHdn514T9QbGs4J0NbQ==", 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow *", 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow *", 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow *", 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.bootstrap-mds", 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQACFqVlk/rdGRAAP2radfhvM0SfOVmxc8HnFg==", 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile bootstrap-mds" 2024-01-15T11:26:04.942 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.bootstrap-mgr", 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQACFqVlDAPeGRAA5/rIuw5UzbpxMgLbY+ftsw==", 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile bootstrap-mgr" 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.bootstrap-osd", 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQACFqVlTwzeGRAA9WvewyeeaFAYnpNrLsoB3g==", 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile bootstrap-osd" 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.943 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.bootstrap-rbd", 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQACFqVllBPeGRAA08XbSPNd4QyRfN3n+JhFOg==", 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile bootstrap-rbd" 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQACFqVl8BreGRAAfbQqX3Q8/yf7dpxU9n7t6w==", 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-01-15T11:26:04.944 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.bootstrap-rgw", 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQACFqVldiLeGRAAtWT1GBTgrrdHg885RtrN0Q==", 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile bootstrap-rgw" 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "mgr.x", 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDgFaVl0fP/IBAAAs2u2ra8ZJ6cSemJuZi/Ww==", 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow *", 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile mgr", 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.945 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.946 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:04.946 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:04.946 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "mgr.y", 2024-01-15T11:26:04.946 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDgFaVl/742HBAAGNipQFrMT9muRnaRJ8ew1A==", 2024-01-15T11:26:04.946 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-15T11:26:04.946 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow *", 2024-01-15T11:26:04.946 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile mgr", 2024-01-15T11:26:04.946 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-15T11:26:04.946 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.946 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:04.946 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-01-15T11:26:04.946 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:26:04.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.934+0000 7fec18af2700 1 -- 172.21.15.113:0/4131461118 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7febf406b350 msgr2=0x7febf406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.934+0000 7fec18af2700 1 --2- 172.21.15.113:0/4131461118 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7febf406b350 0x7febf406d810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fec080097f0 tx=0x7fec08008040 comp rx=0 tx=0).stop 2024-01-15T11:26:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.934+0000 7fec18af2700 1 -- 172.21.15.113:0/4131461118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec1415bed0 msgr2=0x7fec140af9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.934+0000 7fec18af2700 1 --2- 172.21.15.113:0/4131461118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec1415bed0 0x7fec140af9d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fec04014040 tx=0x7fec04004750 comp rx=0 tx=0).stop 2024-01-15T11:26:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.934+0000 7fec18af2700 1 -- 172.21.15.113:0/4131461118 shutdown_connections 2024-01-15T11:26:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.934+0000 7fec18af2700 1 --2- 172.21.15.113:0/4131461118 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7febf406b350 0x7febf406d810 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.934+0000 7fec18af2700 1 --2- 172.21.15.113:0/4131461118 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fec141603c0 0x7fec1406bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.934+0000 7fec18af2700 1 --2- 172.21.15.113:0/4131461118 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fec140d1a50 0x7fec140af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.934+0000 7fec18af2700 1 --2- 172.21.15.113:0/4131461118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec1415bed0 0x7fec140af9d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.934+0000 7fec18af2700 1 -- 172.21.15.113:0/4131461118 >> 172.21.15.113:0/4131461118 conn(0x7fec14073130 msgr2=0x7fec140b0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.935+0000 7fec18af2700 1 -- 172.21.15.113:0/4131461118 shutdown_connections 2024-01-15T11:26:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:04.935+0000 7fec18af2700 1 -- 172.21.15.113:0/4131461118 wait complete. 2024-01-15T11:26:04.949 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-01-15T11:26:05.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.072+0000 7f56617ca700 1 Processor -- start 2024-01-15T11:26:05.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.072+0000 7f56617ca700 1 -- start start 2024-01-15T11:26:05.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.073+0000 7f56617ca700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f565c15ac50 0x7f565c06c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.073+0000 7f56617ca700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f565c06f460 0x7f565c0b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.073+0000 7f56617ca700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c06bf50 con 0x7f565c0b4ef0 2024-01-15T11:26:05.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.073+0000 7f56617ca700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565c0b5380 con 0x7f565c15ac50 2024-01-15T11:26:05.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.073+0000 7f56617ca700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c15b030 con 0x7f565c06f460 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.073+0000 7f565a7fc700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f565c06f460 0x7f565c0b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.074+0000 7f565affd700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f565c15ac50 0x7f565c06c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.074+0000 7f565b7fe700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f565c0b4ef0 0x7f565c0cdf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:55146/0 (socket says 172.21.15.113:55146) 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.074+0000 7f565b7fe700 1 -- 172.21.15.113:0/139874124 learned_addr learned my addr 172.21.15.113:0/139874124 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.074+0000 7f565affd700 1 -- 172.21.15.113:0/139874124 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f565c06f460 msgr2=0x7f565c0b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.074+0000 7f565affd700 1 --2- 172.21.15.113:0/139874124 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f565c06f460 0x7f565c0b49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.074+0000 7f565affd700 1 -- 172.21.15.113:0/139874124 >> v1:172.21.15.141:6789/0 conn(0x7f565c0b4ef0 legacy=0x7f565c0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.074+0000 7f565affd700 1 -- 172.21.15.113:0/139874124 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c06e810 con 0x7f565c15ac50 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.074+0000 7f565affd700 1 --2- 172.21.15.113:0/139874124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f565c15ac50 0x7f565c06c3e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5650009840 tx=0x7f5650009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1b87ff0a8d0c6b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.074+0000 7f5659ffb700 1 -- 172.21.15.113:0/139874124 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650011830 con 0x7f565c15ac50 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.074+0000 7f5659ffb700 1 -- 172.21.15.113:0/139874124 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5650011990 con 0x7f565c15ac50 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.074+0000 7f5659ffb700 1 -- 172.21.15.113:0/139874124 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650011d20 con 0x7f565c15ac50 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.075+0000 7f56617ca700 1 -- 172.21.15.113:0/139874124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f565c15ac50 msgr2=0x7f565c06c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.075+0000 7f56617ca700 1 --2- 172.21.15.113:0/139874124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f565c15ac50 0x7f565c06c3e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5650009840 tx=0x7f5650009b50 comp rx=0 tx=0).stop 2024-01-15T11:26:05.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.075+0000 7f56617ca700 1 -- 172.21.15.113:0/139874124 shutdown_connections 2024-01-15T11:26:05.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.075+0000 7f56617ca700 1 --2- 172.21.15.113:0/139874124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f565c15ac50 0x7f565c06c3e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.075+0000 7f56617ca700 1 --2- 172.21.15.113:0/139874124 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f565c06f460 0x7f565c0b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.075+0000 7f56617ca700 1 -- 172.21.15.113:0/139874124 >> 172.21.15.113:0/139874124 conn(0x7f565c0b3130 msgr2=0x7f565c0d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:05.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.075+0000 7f56617ca700 1 -- 172.21.15.113:0/139874124 shutdown_connections 2024-01-15T11:26:05.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.075+0000 7f56617ca700 1 -- 172.21.15.113:0/139874124 wait complete. 2024-01-15T11:26:05.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.076+0000 7f56617ca700 1 Processor -- start 2024-01-15T11:26:05.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.076+0000 7f56617ca700 1 -- start start 2024-01-15T11:26:05.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.077+0000 7f56617ca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f565c06f460 0x7f565c0c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.077+0000 7f56617ca700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f565c0b4ef0 0x7f565c14ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.077+0000 7f56617ca700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f565c15ac50 0x7f565c0c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.077+0000 7f56617ca700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c150520 con 0x7f565c06f460 2024-01-15T11:26:05.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.077+0000 7f56617ca700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0af5b0 con 0x7f565c15ac50 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.077+0000 7f56617ca700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0af890 con 0x7f565c0b4ef0 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.077+0000 7f565affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f565c06f460 0x7f565c0c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.077+0000 7f565affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f565c06f460 0x7f565c0c5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43380/0 (socket says 172.21.15.113:43380) 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.077+0000 7f565affd700 1 -- 172.21.15.113:0/44146651 learned_addr learned my addr 172.21.15.113:0/44146651 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.077+0000 7f565b7fe700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f565c15ac50 0x7f565c0c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.077+0000 7f565a7fc700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f565c0b4ef0 0x7f565c14ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.078+0000 7f565affd700 1 -- 172.21.15.113:0/44146651 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f565c0b4ef0 msgr2=0x7f565c14ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.078+0000 7f565affd700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f565c0b4ef0 0x7f565c14ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.078+0000 7f565affd700 1 -- 172.21.15.113:0/44146651 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f565c15ac50 msgr2=0x7f565c0c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.078+0000 7f565affd700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f565c15ac50 0x7f565c0c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.078+0000 7f565affd700 1 -- 172.21.15.113:0/44146651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56500094a0 con 0x7f565c06f460 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.078+0000 7f565a7fc700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f565c0b4ef0 0x7f565c14ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.078+0000 7f565affd700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f565c06f460 0x7f565c0c5ac0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f56500090b0 tx=0x7f5650004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.078+0000 7f56437fe700 1 -- 172.21.15.113:0/44146651 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56500036a0 con 0x7f565c06f460 2024-01-15T11:26:05.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.078+0000 7f56437fe700 1 -- 172.21.15.113:0/44146651 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565001e070 con 0x7f565c06f460 2024-01-15T11:26:05.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.078+0000 7f56617ca700 1 -- 172.21.15.113:0/44146651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c0afb70 con 0x7f565c06f460 2024-01-15T11:26:05.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.078+0000 7f56437fe700 1 -- 172.21.15.113:0/44146651 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650003a20 con 0x7f565c06f460 2024-01-15T11:26:05.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.078+0000 7f56617ca700 1 -- 172.21.15.113:0/44146651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c0b0130 con 0x7f565c06f460 2024-01-15T11:26:05.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.079+0000 7f56617ca700 1 -- 172.21.15.113:0/44146651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565c0ce580 con 0x7f565c06f460 2024-01-15T11:26:05.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.079+0000 7f56437fe700 1 -- 172.21.15.113:0/44146651 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5650003b80 con 0x7f565c06f460 2024-01-15T11:26:05.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.080+0000 7f56437fe700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f564806b350 0x7f564806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.080+0000 7f56437fe700 1 -- 172.21.15.113:0/44146651 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f565008a750 con 0x7f565c06f460 2024-01-15T11:26:05.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.080+0000 7f565a7fc700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f564806b350 0x7f564806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.080+0000 7f565a7fc700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f564806b350 0x7f564806d810 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f565c0d01b0 tx=0x7f564c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:05.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.084+0000 7f56437fe700 1 -- 172.21.15.113:0/44146651 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f565c0ce580 con 0x7f565c06f460 2024-01-15T11:26:05.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.355+0000 7f56617ca700 1 -- 172.21.15.113:0/44146651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f565c0ce580 con 0x7f565c06f460 2024-01-15T11:26:05.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.357+0000 7f56437fe700 1 -- 172.21.15.113:0/44146651 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f565c0ce580 con 0x7f565c06f460 2024-01-15T11:26:05.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.359+0000 7f56617ca700 1 -- 172.21.15.113:0/44146651 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f564806b350 msgr2=0x7f564806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.359+0000 7f56617ca700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f564806b350 0x7f564806d810 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f565c0d01b0 tx=0x7f564c008040 comp rx=0 tx=0).stop 2024-01-15T11:26:05.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.359+0000 7f56617ca700 1 -- 172.21.15.113:0/44146651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f565c06f460 msgr2=0x7f565c0c5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.359+0000 7f56617ca700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f565c06f460 0x7f565c0c5ac0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f56500090b0 tx=0x7f5650004750 comp rx=0 tx=0).stop 2024-01-15T11:26:05.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.359+0000 7f56617ca700 1 -- 172.21.15.113:0/44146651 shutdown_connections 2024-01-15T11:26:05.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.359+0000 7f56617ca700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f564806b350 0x7f564806d810 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.360+0000 7f56617ca700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f565c15ac50 0x7f565c0c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.360+0000 7f56617ca700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f565c06f460 0x7f565c0c5ac0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.360+0000 7f56617ca700 1 --2- 172.21.15.113:0/44146651 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f565c0b4ef0 0x7f565c14ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.360+0000 7f56617ca700 1 -- 172.21.15.113:0/44146651 >> 172.21.15.113:0/44146651 conn(0x7f565c0b3130 msgr2=0x7f565c15d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:05.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.360+0000 7f56617ca700 1 -- 172.21.15.113:0/44146651 shutdown_connections 2024-01-15T11:26:05.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.360+0000 7f56617ca700 1 -- 172.21.15.113:0/44146651 wait complete. 2024-01-15T11:26:05.370 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-01-15T11:26:05.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.494+0000 7f3bfec22700 1 Processor -- start 2024-01-15T11:26:05.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.494+0000 7f3bfec22700 1 -- start start 2024-01-15T11:26:05.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfec22700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3bf8073f90 0x7f3bf80ce090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfec22700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3bf806ee50 0x7f3bf8074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfec22700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3bf815c140 0x7f3bf815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfec22700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf8074e90 con 0x7f3bf815c140 2024-01-15T11:26:05.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfec22700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf80ce5d0 con 0x7f3bf8073f90 2024-01-15T11:26:05.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfec22700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80748b0 con 0x7f3bf806ee50 2024-01-15T11:26:05.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfd1bf700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3bf815c140 0x7f3bf815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfd1bf700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3bf815c140 0x7f3bf815f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43390/0 (socket says 172.21.15.113:43390) 2024-01-15T11:26:05.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfd1bf700 1 -- 172.21.15.113:0/934914732 learned_addr learned my addr 172.21.15.113:0/934914732 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:05.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bf7fff700 1 --2- 172.21.15.113:0/934914732 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3bf806ee50 0x7f3bf8074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfd1bf700 1 -- 172.21.15.113:0/934914732 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3bf806ee50 msgr2=0x7f3bf8074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:05.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfc9be700 1 --2- 172.21.15.113:0/934914732 >> v2:172.21.15.141:3300/0 conn(0x7f3bf8073f90 0x7f3bf80ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfd1bf700 1 --2- 172.21.15.113:0/934914732 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3bf806ee50 0x7f3bf8074370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfd1bf700 1 -- 172.21.15.113:0/934914732 >> v2:172.21.15.141:3300/0 conn(0x7f3bf8073f90 msgr2=0x7f3bf80ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:05.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfd1bf700 1 --2- 172.21.15.113:0/934914732 >> v2:172.21.15.141:3300/0 conn(0x7f3bf8073f90 0x7f3bf80ce090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.495+0000 7f3bfd1bf700 1 -- 172.21.15.113:0/934914732 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf815c700 con 0x7f3bf815c140 2024-01-15T11:26:05.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bfd1bf700 1 --2- 172.21.15.113:0/934914732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bf815c140 0x7f3bf815f9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80d1bf0 tx=0x7f3bf000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999a2a7aca4fa1a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:05.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bf77fe700 1 -- 172.21.15.113:0/934914732 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf000e070 con 0x7f3bf815c140 2024-01-15T11:26:05.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bf77fe700 1 -- 172.21.15.113:0/934914732 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf000b9c0 con 0x7f3bf815c140 2024-01-15T11:26:05.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bf77fe700 1 -- 172.21.15.113:0/934914732 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf000bd70 con 0x7f3bf815c140 2024-01-15T11:26:05.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bfec22700 1 -- 172.21.15.113:0/934914732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bf815c140 msgr2=0x7f3bf815f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bfec22700 1 --2- 172.21.15.113:0/934914732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bf815c140 0x7f3bf815f9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80d1bf0 tx=0x7f3bf000b330 comp rx=0 tx=0).stop 2024-01-15T11:26:05.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bfec22700 1 -- 172.21.15.113:0/934914732 shutdown_connections 2024-01-15T11:26:05.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bfec22700 1 --2- 172.21.15.113:0/934914732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bf815c140 0x7f3bf815f9a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bfec22700 1 --2- 172.21.15.113:0/934914732 >> v2:172.21.15.141:3300/0 conn(0x7f3bf8073f90 0x7f3bf80ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bfec22700 1 --2- 172.21.15.113:0/934914732 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3bf806ee50 0x7f3bf8074370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bfec22700 1 -- 172.21.15.113:0/934914732 >> 172.21.15.113:0/934914732 conn(0x7f3bf80cd570 msgr2=0x7f3bf80cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:05.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bfec22700 1 -- 172.21.15.113:0/934914732 shutdown_connections 2024-01-15T11:26:05.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.496+0000 7f3bfec22700 1 -- 172.21.15.113:0/934914732 wait complete. 2024-01-15T11:26:05.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.498+0000 7f3bfec22700 1 Processor -- start 2024-01-15T11:26:05.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.498+0000 7f3bfec22700 1 -- start start 2024-01-15T11:26:05.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfec22700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806ee50 0x7f3bf80cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfec22700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3bf8073f90 0x7f3bf81507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfec22700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bf815c140 0x7f3bf8150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfec22700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf8151220 con 0x7f3bf815c140 2024-01-15T11:26:05.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfec22700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf81e8640 con 0x7f3bf806ee50 2024-01-15T11:26:05.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfec22700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf81e7a30 con 0x7f3bf8073f90 2024-01-15T11:26:05.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfd1bf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bf815c140 0x7f3bf8150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfd1bf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bf815c140 0x7f3bf8150ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43404/0 (socket says 172.21.15.113:43404) 2024-01-15T11:26:05.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfd1bf700 1 -- 172.21.15.113:0/3274563041 learned_addr learned my addr 172.21.15.113:0/3274563041 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:05.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bf7fff700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3bf8073f90 0x7f3bf81507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfc9be700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806ee50 0x7f3bf80cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfd1bf700 1 -- 172.21.15.113:0/3274563041 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3bf8073f90 msgr2=0x7f3bf81507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfd1bf700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3bf8073f90 0x7f3bf81507a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfd1bf700 1 -- 172.21.15.113:0/3274563041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806ee50 msgr2=0x7f3bf80cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfd1bf700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806ee50 0x7f3bf80cf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.499+0000 7f3bfd1bf700 1 -- 172.21.15.113:0/3274563041 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf0009b30 con 0x7f3bf815c140 2024-01-15T11:26:05.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.500+0000 7f3bfd1bf700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bf815c140 0x7f3bf8150ce0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000b370 tx=0x7f3bf0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:05.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.500+0000 7f3bf57fa700 1 -- 172.21.15.113:0/3274563041 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0004310 con 0x7f3bf815c140 2024-01-15T11:26:05.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.500+0000 7f3bf57fa700 1 -- 172.21.15.113:0/3274563041 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf0004470 con 0x7f3bf815c140 2024-01-15T11:26:05.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.500+0000 7f3bfec22700 1 -- 172.21.15.113:0/3274563041 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf81e7d10 con 0x7f3bf815c140 2024-01-15T11:26:05.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.500+0000 7f3bf57fa700 1 -- 172.21.15.113:0/3274563041 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0004820 con 0x7f3bf815c140 2024-01-15T11:26:05.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.500+0000 7f3bfec22700 1 -- 172.21.15.113:0/3274563041 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf81e82d0 con 0x7f3bf815c140 2024-01-15T11:26:05.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.501+0000 7f3bfec22700 1 -- 172.21.15.113:0/3274563041 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf8156240 con 0x7f3bf815c140 2024-01-15T11:26:05.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.501+0000 7f3bf57fa700 1 -- 172.21.15.113:0/3274563041 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3bf001c440 con 0x7f3bf815c140 2024-01-15T11:26:05.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.501+0000 7f3bf57fa700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3bdc06b350 0x7f3bdc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.502+0000 7f3bf57fa700 1 -- 172.21.15.113:0/3274563041 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3bf008ae80 con 0x7f3bf815c140 2024-01-15T11:26:05.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.502+0000 7f3bfc9be700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3bdc06b350 0x7f3bdc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.502+0000 7f3bfc9be700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3bdc06b350 0x7f3bdc06d810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00a990 tx=0x7f3bec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:05.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.506+0000 7f3bf57fa700 1 -- 172.21.15.113:0/3274563041 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bf0056030 con 0x7f3bf815c140 2024-01-15T11:26:05.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.776+0000 7f3bfec22700 1 -- 172.21.15.113:0/3274563041 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f3bf8067f60 con 0x7f3bf815c140 2024-01-15T11:26:05.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.778+0000 7f3bf57fa700 1 -- 172.21.15.113:0/3274563041 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3bf00200a0 con 0x7f3bf815c140 2024-01-15T11:26:05.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.780+0000 7f3bfec22700 1 -- 172.21.15.113:0/3274563041 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3bdc06b350 msgr2=0x7f3bdc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.780+0000 7f3bfec22700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3bdc06b350 0x7f3bdc06d810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00a990 tx=0x7f3bec008040 comp rx=0 tx=0).stop 2024-01-15T11:26:05.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.780+0000 7f3bfec22700 1 -- 172.21.15.113:0/3274563041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bf815c140 msgr2=0x7f3bf8150ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.780+0000 7f3bfec22700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bf815c140 0x7f3bf8150ce0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000b370 tx=0x7f3bf0004ad0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.780+0000 7f3bfec22700 1 -- 172.21.15.113:0/3274563041 shutdown_connections 2024-01-15T11:26:05.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.780+0000 7f3bfec22700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3bdc06b350 0x7f3bdc06d810 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.780+0000 7f3bfec22700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bf815c140 0x7f3bf8150ce0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.780+0000 7f3bfec22700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3bf806ee50 0x7f3bf80cf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.780+0000 7f3bfec22700 1 --2- 172.21.15.113:0/3274563041 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3bf8073f90 0x7f3bf81507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.780+0000 7f3bfec22700 1 -- 172.21.15.113:0/3274563041 >> 172.21.15.113:0/3274563041 conn(0x7f3bf80cd570 msgr2=0x7f3bf806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:05.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.780+0000 7f3bfec22700 1 -- 172.21.15.113:0/3274563041 shutdown_connections 2024-01-15T11:26:05.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.781+0000 7f3bfec22700 1 -- 172.21.15.113:0/3274563041 wait complete. 2024-01-15T11:26:05.791 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-01-15T11:26:05.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.915+0000 7f7c699c4700 1 Processor -- start 2024-01-15T11:26:05.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.915+0000 7f7c699c4700 1 -- start start 2024-01-15T11:26:05.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c699c4700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7c640d3610 0x7f7c640b5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c699c4700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c640d2810 con 0x7f7c6415e480 2024-01-15T11:26:05.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c699c4700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c640b62c0 con 0x7f7c640b5e90 2024-01-15T11:26:05.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c699c4700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c640d2a70 con 0x7f7c640d3610 2024-01-15T11:26:05.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c627fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7c640d3610 0x7f7c640b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c62ffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f7c6415e480 0x7f7c640d2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:43432/0 (socket says 172.21.15.113:43432) 2024-01-15T11:26:05.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c62ffd700 1 -- 172.21.15.113:0/1997382870 learned_addr learned my addr 172.21.15.113:0/1997382870 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:05.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c627fc700 1 -- 172.21.15.113:0/1997382870 >> v1:172.21.15.141:6789/0 conn(0x7f7c640b5e90 legacy=0x7f7c640b7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:05.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c627fc700 1 -- 172.21.15.113:0/1997382870 >> v1:172.21.15.113:6789/0 conn(0x7f7c6415e480 legacy=0x7f7c640d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c627fc700 1 -- 172.21.15.113:0/1997382870 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c640d2d50 con 0x7f7c640d3610 2024-01-15T11:26:05.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c61ffb700 1 -- 172.21.15.113:0/1997382870 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138337539 0 0) 0x7f7c640d2810 con 0x7f7c6415e480 2024-01-15T11:26:05.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c627fc700 1 --2- 172.21.15.113:0/1997382870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c640d3610 0x7f7c640b5950 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7c6408c8e0 tx=0x7f7c54009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234da769711dc6dc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:05.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c61ffb700 1 -- 172.21.15.113:0/1997382870 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54011830 con 0x7f7c640d3610 2024-01-15T11:26:05.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c61ffb700 1 -- 172.21.15.113:0/1997382870 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c54011990 con 0x7f7c640d3610 2024-01-15T11:26:05.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.916+0000 7f7c61ffb700 1 -- 172.21.15.113:0/1997382870 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54011d20 con 0x7f7c640d3610 2024-01-15T11:26:05.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.917+0000 7f7c699c4700 1 -- 172.21.15.113:0/1997382870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c640d3610 msgr2=0x7f7c640b5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.917+0000 7f7c699c4700 1 --2- 172.21.15.113:0/1997382870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c640d3610 0x7f7c640b5950 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7c6408c8e0 tx=0x7f7c54009b20 comp rx=0 tx=0).stop 2024-01-15T11:26:05.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.917+0000 7f7c699c4700 1 -- 172.21.15.113:0/1997382870 shutdown_connections 2024-01-15T11:26:05.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.917+0000 7f7c699c4700 1 --2- 172.21.15.113:0/1997382870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c640d3610 0x7f7c640b5950 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.917+0000 7f7c699c4700 1 -- 172.21.15.113:0/1997382870 >> 172.21.15.113:0/1997382870 conn(0x7f7c640d0950 msgr2=0x7f7c640d0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:05.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.917+0000 7f7c699c4700 1 -- 172.21.15.113:0/1997382870 shutdown_connections 2024-01-15T11:26:05.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.917+0000 7f7c699c4700 1 -- 172.21.15.113:0/1997382870 wait complete. 2024-01-15T11:26:05.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.918+0000 7f7c699c4700 1 Processor -- start 2024-01-15T11:26:05.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.919+0000 7f7c699c4700 1 -- start start 2024-01-15T11:26:05.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c699c4700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c640b5e90 0x7f7c640b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c699c4700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7c640d3610 0x7f7c640c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c699c4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c6415e480 0x7f7c64153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c699c4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c640ca3b0 con 0x7f7c6415e480 2024-01-15T11:26:05.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c699c4700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c640c9e80 con 0x7f7c640b5e90 2024-01-15T11:26:05.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c699c4700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c640ca890 con 0x7f7c640d3610 2024-01-15T11:26:05.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c637fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c6415e480 0x7f7c64153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c637fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c6415e480 0x7f7c64153a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43428/0 (socket says 172.21.15.113:43428) 2024-01-15T11:26:05.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c637fe700 1 -- 172.21.15.113:0/3745110363 learned_addr learned my addr 172.21.15.113:0/3745110363 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:05.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c62ffd700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c640b5e90 0x7f7c640b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c627fc700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7c640d3610 0x7f7c640c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c637fe700 1 -- 172.21.15.113:0/3745110363 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7c640d3610 msgr2=0x7f7c640c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c637fe700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7c640d3610 0x7f7c640c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c637fe700 1 -- 172.21.15.113:0/3745110363 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c640b5e90 msgr2=0x7f7c640b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:05.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c637fe700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c640b5e90 0x7f7c640b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:05.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c637fe700 1 -- 172.21.15.113:0/3745110363 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c540094a0 con 0x7f7c6415e480 2024-01-15T11:26:05.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c62ffd700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c640b5e90 0x7f7c640b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:05.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c627fc700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7c640d3610 0x7f7c640c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:05.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c637fe700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c6415e480 0x7f7c64153a80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00c7b0 tx=0x7f7c5c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:05.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c4bfff700 1 -- 172.21.15.113:0/3745110363 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c003e10 con 0x7f7c6415e480 2024-01-15T11:26:05.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c4bfff700 1 -- 172.21.15.113:0/3745110363 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5c0050f0 con 0x7f7c6415e480 2024-01-15T11:26:05.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c699c4700 1 -- 172.21.15.113:0/3745110363 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c640ca0d0 con 0x7f7c6415e480 2024-01-15T11:26:05.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c4bfff700 1 -- 172.21.15.113:0/3745110363 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0054a0 con 0x7f7c6415e480 2024-01-15T11:26:05.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.920+0000 7f7c699c4700 1 -- 172.21.15.113:0/3745110363 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6415ece0 con 0x7f7c6415e480 2024-01-15T11:26:05.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.921+0000 7f7c699c4700 1 -- 172.21.15.113:0/3745110363 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6415e860 con 0x7f7c6415e480 2024-01-15T11:26:05.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.922+0000 7f7c4bfff700 1 -- 172.21.15.113:0/3745110363 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7c5c005600 con 0x7f7c6415e480 2024-01-15T11:26:05.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.922+0000 7f7c4bfff700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7c5006b350 0x7f7c5006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:05.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.922+0000 7f7c4bfff700 1 -- 172.21.15.113:0/3745110363 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7c5c088aa0 con 0x7f7c6415e480 2024-01-15T11:26:05.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.922+0000 7f7c62ffd700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7c5006b350 0x7f7c5006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:05.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.923+0000 7f7c62ffd700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7c5006b350 0x7f7c5006d810 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7c58000f60 tx=0x7f7c58007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:05.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:05.927+0000 7f7c4bfff700 1 -- 172.21.15.113:0/3745110363 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c6415e860 con 0x7f7c6415e480 2024-01-15T11:26:06.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.199+0000 7f7c699c4700 1 -- 172.21.15.113:0/3745110363 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f7c6415e860 con 0x7f7c6415e480 2024-01-15T11:26:06.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.201+0000 7f7c4bfff700 1 -- 172.21.15.113:0/3745110363 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7c6415e860 con 0x7f7c6415e480 2024-01-15T11:26:06.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.203+0000 7f7c699c4700 1 -- 172.21.15.113:0/3745110363 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7c5006b350 msgr2=0x7f7c5006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:06.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.203+0000 7f7c699c4700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7c5006b350 0x7f7c5006d810 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7c58000f60 tx=0x7f7c58007d30 comp rx=0 tx=0).stop 2024-01-15T11:26:06.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.203+0000 7f7c699c4700 1 -- 172.21.15.113:0/3745110363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c6415e480 msgr2=0x7f7c64153a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:06.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.203+0000 7f7c699c4700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c6415e480 0x7f7c64153a80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00c7b0 tx=0x7f7c5c00cb70 comp rx=0 tx=0).stop 2024-01-15T11:26:06.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.203+0000 7f7c699c4700 1 -- 172.21.15.113:0/3745110363 shutdown_connections 2024-01-15T11:26:06.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.203+0000 7f7c699c4700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7c5006b350 0x7f7c5006d810 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.203+0000 7f7c699c4700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c6415e480 0x7f7c64153a80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.203+0000 7f7c699c4700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7c640b5e90 0x7f7c640b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.203+0000 7f7c699c4700 1 --2- 172.21.15.113:0/3745110363 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7c640d3610 0x7f7c640c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.203+0000 7f7c699c4700 1 -- 172.21.15.113:0/3745110363 >> 172.21.15.113:0/3745110363 conn(0x7f7c640d0950 msgr2=0x7f7c64061a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:06.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.203+0000 7f7c699c4700 1 -- 172.21.15.113:0/3745110363 shutdown_connections 2024-01-15T11:26:06.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.203+0000 7f7c699c4700 1 -- 172.21.15.113:0/3745110363 wait complete. 2024-01-15T11:26:06.213 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-01-15T11:26:06.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.337+0000 7efe863b4700 1 Processor -- start 2024-01-15T11:26:06.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.337+0000 7efe863b4700 1 -- start start 2024-01-15T11:26:06.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.338+0000 7efe863b4700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7efe800b19f0 0x7efe801603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:06.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.338+0000 7efe863b4700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe800cd100 con 0x7efe8015bed0 2024-01-15T11:26:06.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.338+0000 7efe863b4700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe800d09a0 con 0x7efe800d2160 2024-01-15T11:26:06.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.338+0000 7efe863b4700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe800b1e80 con 0x7efe800b19f0 2024-01-15T11:26:06.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.338+0000 7efe7ffff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7efe8015bed0 0x7efe800d1a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:43444/0 (socket says 172.21.15.113:43444) 2024-01-15T11:26:06.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.338+0000 7efe7ffff700 1 -- 172.21.15.113:0/3714167134 learned_addr learned my addr 172.21.15.113:0/3714167134 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:06.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.338+0000 7efe84951700 1 --2- 172.21.15.113:0/3714167134 >> v2:172.21.15.141:3300/0 conn(0x7efe800b19f0 0x7efe801603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:06.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.338+0000 7efe7effd700 1 -- 172.21.15.113:0/3714167134 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228547559 0 0) 0x7efe800cd100 con 0x7efe8015bed0 2024-01-15T11:26:06.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.338+0000 7efe7effd700 1 -- 172.21.15.113:0/3714167134 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe680035b0 con 0x7efe8015bed0 2024-01-15T11:26:06.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.339+0000 7efe7effd700 1 -- 172.21.15.113:0/3714167134 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 696176973 0 0) 0x7efe680035b0 con 0x7efe8015bed0 2024-01-15T11:26:06.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.339+0000 7efe84951700 1 -- 172.21.15.113:0/3714167134 >> v1:172.21.15.141:6789/0 conn(0x7efe800d2160 legacy=0x7efe800cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:06.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.339+0000 7efe84951700 1 -- 172.21.15.113:0/3714167134 >> v1:172.21.15.113:6789/0 conn(0x7efe8015bed0 legacy=0x7efe800d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:06.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.339+0000 7efe84951700 1 -- 172.21.15.113:0/3714167134 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe800cc7d0 con 0x7efe800b19f0 2024-01-15T11:26:06.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.339+0000 7efe84951700 1 --2- 172.21.15.113:0/3714167134 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efe800b19f0 0x7efe801603c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efe78009d90 tx=0x7efe7800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31f8f7a8d8b6dc4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:06.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.339+0000 7efe7effd700 1 -- 172.21.15.113:0/3714167134 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7800e070 con 0x7efe800b19f0 2024-01-15T11:26:06.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.339+0000 7efe7effd700 1 -- 172.21.15.113:0/3714167134 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe78003a20 con 0x7efe800b19f0 2024-01-15T11:26:06.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.339+0000 7efe7effd700 1 -- 172.21.15.113:0/3714167134 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe70001670 con 0x7efe800b19f0 2024-01-15T11:26:06.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.339+0000 7efe863b4700 1 -- 172.21.15.113:0/3714167134 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efe800b19f0 msgr2=0x7efe801603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:06.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.339+0000 7efe863b4700 1 --2- 172.21.15.113:0/3714167134 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efe800b19f0 0x7efe801603c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efe78009d90 tx=0x7efe7800b330 comp rx=0 tx=0).stop 2024-01-15T11:26:06.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.340+0000 7efe863b4700 1 -- 172.21.15.113:0/3714167134 shutdown_connections 2024-01-15T11:26:06.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.340+0000 7efe863b4700 1 --2- 172.21.15.113:0/3714167134 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efe800b19f0 0x7efe801603c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.340+0000 7efe863b4700 1 -- 172.21.15.113:0/3714167134 >> 172.21.15.113:0/3714167134 conn(0x7efe80073130 msgr2=0x7efe8009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:06.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.340+0000 7efe863b4700 1 -- 172.21.15.113:0/3714167134 shutdown_connections 2024-01-15T11:26:06.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.340+0000 7efe863b4700 1 -- 172.21.15.113:0/3714167134 wait complete. 2024-01-15T11:26:06.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.341+0000 7efe863b4700 1 Processor -- start 2024-01-15T11:26:06.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.341+0000 7efe863b4700 1 -- start start 2024-01-15T11:26:06.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.342+0000 7efe863b4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe800b19f0 0x7efe80158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:06.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.342+0000 7efe863b4700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efe800d2160 0x7efe800b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:06.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.342+0000 7efe863b4700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efe8015bed0 0x7efe800ce900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:06.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.342+0000 7efe863b4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe800b6b40 con 0x7efe800b19f0 2024-01-15T11:26:06.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.342+0000 7efe863b4700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe801592e0 con 0x7efe800d2160 2024-01-15T11:26:06.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.342+0000 7efe863b4700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe800b6d70 con 0x7efe8015bed0 2024-01-15T11:26:06.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.342+0000 7efe7ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe800b19f0 0x7efe80158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:06.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.342+0000 7efe7ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe800b19f0 0x7efe80158da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43444/0 (socket says 172.21.15.113:43444) 2024-01-15T11:26:06.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.342+0000 7efe7ffff700 1 -- 172.21.15.113:0/893825499 learned_addr learned my addr 172.21.15.113:0/893825499 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:06.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.343+0000 7efe7ffff700 1 -- 172.21.15.113:0/893825499 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efe8015bed0 msgr2=0x7efe800ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:06.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.343+0000 7efe7ffff700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efe8015bed0 0x7efe800ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.343+0000 7efe7ffff700 1 -- 172.21.15.113:0/893825499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efe800d2160 msgr2=0x7efe800b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:06.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.343+0000 7efe7f7fe700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efe800d2160 0x7efe800b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:06.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.343+0000 7efe7ffff700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efe800d2160 0x7efe800b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.343+0000 7efe7ffff700 1 -- 172.21.15.113:0/893825499 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe78009a70 con 0x7efe800b19f0 2024-01-15T11:26:06.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.343+0000 7efe7ffff700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe800b19f0 0x7efe80158da0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efe7000c520 tx=0x7efe7000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:06.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.343+0000 7efe7cff9700 1 -- 172.21.15.113:0/893825499 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe70004c80 con 0x7efe800b19f0 2024-01-15T11:26:06.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.343+0000 7efe7cff9700 1 -- 172.21.15.113:0/893825499 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe70004e20 con 0x7efe800b19f0 2024-01-15T11:26:06.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.343+0000 7efe863b4700 1 -- 172.21.15.113:0/893825499 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe800b7050 con 0x7efe800b19f0 2024-01-15T11:26:06.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.343+0000 7efe7cff9700 1 -- 172.21.15.113:0/893825499 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe70007310 con 0x7efe800b19f0 2024-01-15T11:26:06.347 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.343+0000 7efe863b4700 1 -- 172.21.15.113:0/893825499 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe800b7610 con 0x7efe800b19f0 2024-01-15T11:26:06.347 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.344+0000 7efe863b4700 1 -- 172.21.15.113:0/893825499 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe80067f60 con 0x7efe800b19f0 2024-01-15T11:26:06.347 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.344+0000 7efe7cff9700 1 -- 172.21.15.113:0/893825499 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efe70007470 con 0x7efe800b19f0 2024-01-15T11:26:06.347 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.345+0000 7efe7cff9700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efe6c06b350 0x7efe6c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:06.347 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.345+0000 7efe7cff9700 1 -- 172.21.15.113:0/893825499 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efe700896c0 con 0x7efe800b19f0 2024-01-15T11:26:06.347 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.345+0000 7efe7f7fe700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efe6c06b350 0x7efe6c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:06.347 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.345+0000 7efe7f7fe700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efe6c06b350 0x7efe6c06d810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efe801f49d0 tx=0x7efe74009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:06.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.349+0000 7efe7cff9700 1 -- 172.21.15.113:0/893825499 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe70054880 con 0x7efe800b19f0 2024-01-15T11:26:06.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.617+0000 7efe863b4700 1 -- 172.21.15.113:0/893825499 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7efe800b7bd0 con 0x7efe800b19f0 2024-01-15T11:26:06.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.621+0000 7efe7cff9700 1 -- 172.21.15.113:0/893825499 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7efe70057ea0 con 0x7efe800b19f0 2024-01-15T11:26:06.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.623+0000 7efe863b4700 1 -- 172.21.15.113:0/893825499 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efe6c06b350 msgr2=0x7efe6c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:06.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.623+0000 7efe863b4700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efe6c06b350 0x7efe6c06d810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efe801f49d0 tx=0x7efe74009040 comp rx=0 tx=0).stop 2024-01-15T11:26:06.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.623+0000 7efe863b4700 1 -- 172.21.15.113:0/893825499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe800b19f0 msgr2=0x7efe80158da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:06.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.623+0000 7efe863b4700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe800b19f0 0x7efe80158da0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efe7000c520 tx=0x7efe7000c8e0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.623+0000 7efe863b4700 1 -- 172.21.15.113:0/893825499 shutdown_connections 2024-01-15T11:26:06.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.623+0000 7efe863b4700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7efe6c06b350 0x7efe6c06d810 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.623+0000 7efe863b4700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7efe8015bed0 0x7efe800ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.623+0000 7efe863b4700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe800b19f0 0x7efe80158da0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.623+0000 7efe863b4700 1 --2- 172.21.15.113:0/893825499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7efe800d2160 0x7efe800b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.623+0000 7efe863b4700 1 -- 172.21.15.113:0/893825499 >> 172.21.15.113:0/893825499 conn(0x7efe80073130 msgr2=0x7efe800b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:06.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.623+0000 7efe863b4700 1 -- 172.21.15.113:0/893825499 shutdown_connections 2024-01-15T11:26:06.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.623+0000 7efe863b4700 1 -- 172.21.15.113:0/893825499 wait complete. 2024-01-15T11:26:06.634 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-01-15T11:26:06.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.757+0000 7fca25da3700 1 Processor -- start 2024-01-15T11:26:06.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.757+0000 7fca25da3700 1 -- start start 2024-01-15T11:26:06.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca25da3700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fca20158670 0x7fca200622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:06.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca25da3700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fca20158a50 0x7fca200c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:06.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca25da3700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fca20162a30 0x7fca200c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:06.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca25da3700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca200cfff0 con 0x7fca20162a30 2024-01-15T11:26:06.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca25da3700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca200cfd10 con 0x7fca20158670 2024-01-15T11:26:06.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca25da3700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca20161f50 con 0x7fca20158a50 2024-01-15T11:26:06.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1ffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fca20162a30 0x7fca200c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:06.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1ffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fca20162a30 0x7fca200c9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43456/0 (socket says 172.21.15.113:43456) 2024-01-15T11:26:06.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1ffff700 1 -- 172.21.15.113:0/3525631986 learned_addr learned my addr 172.21.15.113:0/3525631986 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:06.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1f7fe700 1 --2- 172.21.15.113:0/3525631986 >> v2:172.21.15.141:3300/0 conn(0x7fca20158670 0x7fca200622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:06.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1effd700 1 --2- 172.21.15.113:0/3525631986 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fca20158a50 0x7fca200c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:06.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1ffff700 1 -- 172.21.15.113:0/3525631986 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fca20158a50 msgr2=0x7fca200c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:06.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1ffff700 1 --2- 172.21.15.113:0/3525631986 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fca20158a50 0x7fca200c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1ffff700 1 -- 172.21.15.113:0/3525631986 >> v2:172.21.15.141:3300/0 conn(0x7fca20158670 msgr2=0x7fca200622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:06.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1ffff700 1 --2- 172.21.15.113:0/3525631986 >> v2:172.21.15.141:3300/0 conn(0x7fca20158670 0x7fca200622a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1ffff700 1 -- 172.21.15.113:0/3525631986 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca20162230 con 0x7fca20162a30 2024-01-15T11:26:06.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1f7fe700 1 --2- 172.21.15.113:0/3525631986 >> v2:172.21.15.141:3300/0 conn(0x7fca20158670 0x7fca200622a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:06.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1effd700 1 --2- 172.21.15.113:0/3525631986 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fca20158a50 0x7fca200c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:06.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1ffff700 1 --2- 172.21.15.113:0/3525631986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca20162a30 0x7fca200c9df0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fca2009e5c0 tx=0x7fca1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d56caab24fb016 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:06.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1e7fc700 1 -- 172.21.15.113:0/3525631986 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1800e070 con 0x7fca20162a30 2024-01-15T11:26:06.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1e7fc700 1 -- 172.21.15.113:0/3525631986 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1800b9c0 con 0x7fca20162a30 2024-01-15T11:26:06.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.758+0000 7fca1e7fc700 1 -- 172.21.15.113:0/3525631986 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1800bd70 con 0x7fca20162a30 2024-01-15T11:26:06.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.759+0000 7fca25da3700 1 -- 172.21.15.113:0/3525631986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca20162a30 msgr2=0x7fca200c9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:06.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.759+0000 7fca25da3700 1 --2- 172.21.15.113:0/3525631986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca20162a30 0x7fca200c9df0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fca2009e5c0 tx=0x7fca1800b330 comp rx=0 tx=0).stop 2024-01-15T11:26:06.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.759+0000 7fca25da3700 1 -- 172.21.15.113:0/3525631986 shutdown_connections 2024-01-15T11:26:06.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.759+0000 7fca25da3700 1 --2- 172.21.15.113:0/3525631986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca20162a30 0x7fca200c9df0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.759+0000 7fca25da3700 1 --2- 172.21.15.113:0/3525631986 >> v2:172.21.15.141:3300/0 conn(0x7fca20158670 0x7fca200622a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.759+0000 7fca25da3700 1 --2- 172.21.15.113:0/3525631986 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fca20158a50 0x7fca200c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.759+0000 7fca25da3700 1 -- 172.21.15.113:0/3525631986 >> 172.21.15.113:0/3525631986 conn(0x7fca200b57b0 msgr2=0x7fca200b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:06.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.759+0000 7fca25da3700 1 -- 172.21.15.113:0/3525631986 shutdown_connections 2024-01-15T11:26:06.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.759+0000 7fca25da3700 1 -- 172.21.15.113:0/3525631986 wait complete. 2024-01-15T11:26:06.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.760+0000 7fca25da3700 1 Processor -- start 2024-01-15T11:26:06.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.761+0000 7fca25da3700 1 -- start start 2024-01-15T11:26:06.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.761+0000 7fca25da3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca20158670 0x7fca200b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:06.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca25da3700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fca20158a50 0x7fca201555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:06.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca25da3700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca20162a30 0x7fca2015ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:06.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca25da3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca200b50b0 con 0x7fca20158670 2024-01-15T11:26:06.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca25da3700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca200b4430 con 0x7fca20162a30 2024-01-15T11:26:06.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca25da3700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca200b4710 con 0x7fca20158a50 2024-01-15T11:26:06.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca1f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca20158670 0x7fca200b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:06.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca1f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca20158670 0x7fca200b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43468/0 (socket says 172.21.15.113:43468) 2024-01-15T11:26:06.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca1f7fe700 1 -- 172.21.15.113:0/1455569354 learned_addr learned my addr 172.21.15.113:0/1455569354 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:06.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca1effd700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fca20158a50 0x7fca201555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:06.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca1ffff700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca20162a30 0x7fca2015ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:06.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca1f7fe700 1 -- 172.21.15.113:0/1455569354 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fca20158a50 msgr2=0x7fca201555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:06.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca1f7fe700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fca20158a50 0x7fca201555f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca1f7fe700 1 -- 172.21.15.113:0/1455569354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca20162a30 msgr2=0x7fca2015ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:06.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca1f7fe700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca20162a30 0x7fca2015ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:06.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca1f7fe700 1 -- 172.21.15.113:0/1455569354 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca18009b30 con 0x7fca20158670 2024-01-15T11:26:06.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca1f7fe700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca20158670 0x7fca200b3b50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fca1000c740 tx=0x7fca1000ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:06.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca07fff700 1 -- 172.21.15.113:0/1455569354 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10011800 con 0x7fca20158670 2024-01-15T11:26:06.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca07fff700 1 -- 172.21.15.113:0/1455569354 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca10011960 con 0x7fca20158670 2024-01-15T11:26:06.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca07fff700 1 -- 172.21.15.113:0/1455569354 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10011cd0 con 0x7fca20158670 2024-01-15T11:26:06.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca25da3700 1 -- 172.21.15.113:0/1455569354 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca200b49f0 con 0x7fca20158670 2024-01-15T11:26:06.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.762+0000 7fca25da3700 1 -- 172.21.15.113:0/1455569354 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca200b54d0 con 0x7fca20158670 2024-01-15T11:26:06.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.763+0000 7fca25da3700 1 -- 172.21.15.113:0/1455569354 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca20067f60 con 0x7fca20158670 2024-01-15T11:26:06.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.764+0000 7fca07fff700 1 -- 172.21.15.113:0/1455569354 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fca10021350 con 0x7fca20158670 2024-01-15T11:26:06.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.764+0000 7fca07fff700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fca0c06b350 0x7fca0c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:06.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.764+0000 7fca07fff700 1 -- 172.21.15.113:0/1455569354 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fca10089590 con 0x7fca20158670 2024-01-15T11:26:06.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.764+0000 7fca1effd700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fca0c06b350 0x7fca0c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:06.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.765+0000 7fca1effd700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fca0c06b350 0x7fca0c06d810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fca2015f7d0 tx=0x7fca14008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:06.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:06.769+0000 7fca07fff700 1 -- 172.21.15.113:0/1455569354 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca100547e0 con 0x7fca20158670 2024-01-15T11:26:07.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.039+0000 7fca25da3700 1 -- 172.21.15.113:0/1455569354 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fca20074a20 con 0x7fca20158670 2024-01-15T11:26:07.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.043+0000 7fca07fff700 1 -- 172.21.15.113:0/1455569354 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7fca10057e00 con 0x7fca20158670 2024-01-15T11:26:07.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.045+0000 7fca25da3700 1 -- 172.21.15.113:0/1455569354 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fca0c06b350 msgr2=0x7fca0c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.045+0000 7fca25da3700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fca0c06b350 0x7fca0c06d810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fca2015f7d0 tx=0x7fca14008040 comp rx=0 tx=0).stop 2024-01-15T11:26:07.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.045+0000 7fca25da3700 1 -- 172.21.15.113:0/1455569354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca20158670 msgr2=0x7fca200b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.045+0000 7fca25da3700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca20158670 0x7fca200b3b50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fca1000c740 tx=0x7fca1000ca50 comp rx=0 tx=0).stop 2024-01-15T11:26:07.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.045+0000 7fca25da3700 1 -- 172.21.15.113:0/1455569354 shutdown_connections 2024-01-15T11:26:07.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.045+0000 7fca25da3700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fca0c06b350 0x7fca0c06d810 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.045+0000 7fca25da3700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca20162a30 0x7fca2015ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.045+0000 7fca25da3700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca20158670 0x7fca200b3b50 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.045+0000 7fca25da3700 1 --2- 172.21.15.113:0/1455569354 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fca20158a50 0x7fca201555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.045+0000 7fca25da3700 1 -- 172.21.15.113:0/1455569354 >> 172.21.15.113:0/1455569354 conn(0x7fca200b57b0 msgr2=0x7fca200ce1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:07.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.045+0000 7fca25da3700 1 -- 172.21.15.113:0/1455569354 shutdown_connections 2024-01-15T11:26:07.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.045+0000 7fca25da3700 1 -- 172.21.15.113:0/1455569354 wait complete. 2024-01-15T11:26:07.056 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-01-15T11:26:07.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.180+0000 7f0c09484700 1 Processor -- start 2024-01-15T11:26:07.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.180+0000 7f0c09484700 1 -- start start 2024-01-15T11:26:07.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c09484700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0c040d0870 0x7f0c040d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c09484700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0c040b6ef0 0x7f0c040b7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c09484700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c040d20b0 con 0x7f0c0415e1e0 2024-01-15T11:26:07.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c09484700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c040d3150 con 0x7f0c040b6ef0 2024-01-15T11:26:07.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c09484700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c040d1240 con 0x7f0c040d0870 2024-01-15T11:26:07.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c037fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0c040b6ef0 0x7f0c040b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c02ffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f0c0415e1e0 0x7f0c040d34c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:43446/0 (socket says 172.21.15.113:43446) 2024-01-15T11:26:07.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c02ffd700 1 -- 172.21.15.113:0/3247506560 learned_addr learned my addr 172.21.15.113:0/3247506560 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:07.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c027fc700 1 --2- 172.21.15.113:0/3247506560 >> v2:172.21.15.141:3300/0 conn(0x7f0c040d0870 0x7f0c040d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c037fe700 1 -- 172.21.15.113:0/3247506560 >> v2:172.21.15.141:3300/0 conn(0x7f0c040d0870 msgr2=0x7f0c040d0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:07.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c037fe700 1 --2- 172.21.15.113:0/3247506560 >> v2:172.21.15.141:3300/0 conn(0x7f0c040d0870 0x7f0c040d0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c037fe700 1 -- 172.21.15.113:0/3247506560 >> v1:172.21.15.113:6789/0 conn(0x7f0c0415e1e0 legacy=0x7f0c040d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c037fe700 1 -- 172.21.15.113:0/3247506560 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c040d2390 con 0x7f0c040b6ef0 2024-01-15T11:26:07.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c01ffb700 1 -- 172.21.15.113:0/3247506560 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263667310 0 0) 0x7f0c040d20b0 con 0x7f0c0415e1e0 2024-01-15T11:26:07.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.181+0000 7f0c037fe700 1 --2- 172.21.15.113:0/3247506560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c040b6ef0 0x7f0c040b7360 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0c04161ba0 tx=0x7f0bfc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee42439947c2dd67 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:07.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.182+0000 7f0c01ffb700 1 -- 172.21.15.113:0/3247506560 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc00e070 con 0x7f0c040b6ef0 2024-01-15T11:26:07.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.182+0000 7f0c01ffb700 1 -- 172.21.15.113:0/3247506560 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc00bbd0 con 0x7f0c040b6ef0 2024-01-15T11:26:07.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.182+0000 7f0c01ffb700 1 -- 172.21.15.113:0/3247506560 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc012430 con 0x7f0c040b6ef0 2024-01-15T11:26:07.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.182+0000 7f0c09484700 1 -- 172.21.15.113:0/3247506560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c040b6ef0 msgr2=0x7f0c040b7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.182+0000 7f0c09484700 1 --2- 172.21.15.113:0/3247506560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c040b6ef0 0x7f0c040b7360 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0c04161ba0 tx=0x7f0bfc00b330 comp rx=0 tx=0).stop 2024-01-15T11:26:07.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.182+0000 7f0c09484700 1 -- 172.21.15.113:0/3247506560 shutdown_connections 2024-01-15T11:26:07.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.182+0000 7f0c09484700 1 --2- 172.21.15.113:0/3247506560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c040b6ef0 0x7f0c040b7360 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.182+0000 7f0c09484700 1 --2- 172.21.15.113:0/3247506560 >> v2:172.21.15.141:3300/0 conn(0x7f0c040d0870 0x7f0c040d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.182+0000 7f0c09484700 1 -- 172.21.15.113:0/3247506560 >> 172.21.15.113:0/3247506560 conn(0x7f0c040b5b50 msgr2=0x7f0c040b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:07.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.182+0000 7f0c09484700 1 -- 172.21.15.113:0/3247506560 shutdown_connections 2024-01-15T11:26:07.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.182+0000 7f0c09484700 1 -- 172.21.15.113:0/3247506560 wait complete. 2024-01-15T11:26:07.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.184+0000 7f0c09484700 1 Processor -- start 2024-01-15T11:26:07.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.184+0000 7f0c09484700 1 -- start start 2024-01-15T11:26:07.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c09484700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c040b6ef0 0x7f0c041526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c09484700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c040d0870 0x7f0c040ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c09484700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c0415e1e0 0x7f0c040ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c09484700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c041f7030 con 0x7f0c040b6ef0 2024-01-15T11:26:07.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c09484700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c041f7bb0 con 0x7f0c0415e1e0 2024-01-15T11:26:07.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c09484700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c041f7280 con 0x7f0c040d0870 2024-01-15T11:26:07.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c02ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c040b6ef0 0x7f0c041526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c02ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c040b6ef0 0x7f0c041526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43472/0 (socket says 172.21.15.113:43472) 2024-01-15T11:26:07.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c02ffd700 1 -- 172.21.15.113:0/569521636 learned_addr learned my addr 172.21.15.113:0/569521636 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:07.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c02ffd700 1 -- 172.21.15.113:0/569521636 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c040d0870 msgr2=0x7f0c040ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c02ffd700 1 --2- 172.21.15.113:0/569521636 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c040d0870 0x7f0c040ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c02ffd700 1 -- 172.21.15.113:0/569521636 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c0415e1e0 msgr2=0x7f0c040ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c02ffd700 1 --2- 172.21.15.113:0/569521636 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c0415e1e0 0x7f0c040ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c02ffd700 1 -- 172.21.15.113:0/569521636 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc009a70 con 0x7f0c040b6ef0 2024-01-15T11:26:07.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c02ffd700 1 --2- 172.21.15.113:0/569521636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c040b6ef0 0x7f0c041526e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800c580 tx=0x7f0bf800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:07.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0beb7fe700 1 -- 172.21.15.113:0/569521636 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8004c80 con 0x7f0c040b6ef0 2024-01-15T11:26:07.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0beb7fe700 1 -- 172.21.15.113:0/569521636 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf8004e00 con 0x7f0c040b6ef0 2024-01-15T11:26:07.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0c09484700 1 -- 172.21.15.113:0/569521636 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c041f7560 con 0x7f0c040b6ef0 2024-01-15T11:26:07.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.185+0000 7f0beb7fe700 1 -- 172.21.15.113:0/569521636 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8007310 con 0x7f0c040b6ef0 2024-01-15T11:26:07.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.186+0000 7f0c09484700 1 -- 172.21.15.113:0/569521636 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c04152c20 con 0x7f0c040b6ef0 2024-01-15T11:26:07.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.186+0000 7f0c09484700 1 -- 172.21.15.113:0/569521636 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c040751e0 con 0x7f0c040b6ef0 2024-01-15T11:26:07.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.187+0000 7f0beb7fe700 1 -- 172.21.15.113:0/569521636 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0bf8007470 con 0x7f0c040b6ef0 2024-01-15T11:26:07.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.187+0000 7f0beb7fe700 1 --2- 172.21.15.113:0/569521636 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0bf006b350 0x7f0bf006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.187+0000 7f0beb7fe700 1 -- 172.21.15.113:0/569521636 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0bf8089540 con 0x7f0c040b6ef0 2024-01-15T11:26:07.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.187+0000 7f0c027fc700 1 --2- 172.21.15.113:0/569521636 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0bf006b350 0x7f0bf006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.188+0000 7f0c027fc700 1 --2- 172.21.15.113:0/569521636 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0bf006b350 0x7f0bf006d810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0c040cc560 tx=0x7f0bf4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:07.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.192+0000 7f0beb7fe700 1 -- 172.21.15.113:0/569521636 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bf80546f0 con 0x7f0c040b6ef0 2024-01-15T11:26:07.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.457+0000 7f0c09484700 1 -- 172.21.15.113:0/569521636 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f0c040b9360 con 0x7f0c040b6ef0 2024-01-15T11:26:07.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.461+0000 7f0beb7fe700 1 -- 172.21.15.113:0/569521636 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0bf8057d10 con 0x7f0c040b6ef0 2024-01-15T11:26:07.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.464+0000 7f0c09484700 1 -- 172.21.15.113:0/569521636 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0bf006b350 msgr2=0x7f0bf006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.464+0000 7f0c09484700 1 --2- 172.21.15.113:0/569521636 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0bf006b350 0x7f0bf006d810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0c040cc560 tx=0x7f0bf4008040 comp rx=0 tx=0).stop 2024-01-15T11:26:07.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.464+0000 7f0c09484700 1 -- 172.21.15.113:0/569521636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c040b6ef0 msgr2=0x7f0c041526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.464+0000 7f0c09484700 1 --2- 172.21.15.113:0/569521636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c040b6ef0 0x7f0c041526e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800c580 tx=0x7f0bf800c940 comp rx=0 tx=0).stop 2024-01-15T11:26:07.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.464+0000 7f0c09484700 1 -- 172.21.15.113:0/569521636 shutdown_connections 2024-01-15T11:26:07.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.464+0000 7f0c09484700 1 --2- 172.21.15.113:0/569521636 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0bf006b350 0x7f0bf006d810 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.464+0000 7f0c09484700 1 --2- 172.21.15.113:0/569521636 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c0415e1e0 0x7f0c040ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.464+0000 7f0c09484700 1 --2- 172.21.15.113:0/569521636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c040b6ef0 0x7f0c041526e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.464+0000 7f0c09484700 1 --2- 172.21.15.113:0/569521636 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c040d0870 0x7f0c040ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.464+0000 7f0c09484700 1 -- 172.21.15.113:0/569521636 >> 172.21.15.113:0/569521636 conn(0x7f0c040b5b50 msgr2=0x7f0c0406b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:07.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.464+0000 7f0c09484700 1 -- 172.21.15.113:0/569521636 shutdown_connections 2024-01-15T11:26:07.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.464+0000 7f0c09484700 1 -- 172.21.15.113:0/569521636 wait complete. 2024-01-15T11:26:07.475 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:26:07.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.599+0000 7ff765e91700 1 Processor -- start 2024-01-15T11:26:07.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.599+0000 7ff765e91700 1 -- start start 2024-01-15T11:26:07.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.599+0000 7ff765e91700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff760147780 0x7ff76015e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.599+0000 7ff765e91700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff76006dc20 0x7ff760073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.599+0000 7ff765e91700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff76015eb00 con 0x7ff7601473a0 2024-01-15T11:26:07.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.599+0000 7ff765e91700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76015ede0 con 0x7ff76006dc20 2024-01-15T11:26:07.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.599+0000 7ff765e91700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76015f0c0 con 0x7ff760147780 2024-01-15T11:26:07.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.600+0000 7ff75ffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff76006dc20 0x7ff760073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.600+0000 7ff75ffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff76006dc20 0x7ff760073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43488/0 (socket says 172.21.15.113:43488) 2024-01-15T11:26:07.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.600+0000 7ff75ffff700 1 -- 172.21.15.113:0/980921509 learned_addr learned my addr 172.21.15.113:0/980921509 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:07.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.600+0000 7ff75effd700 1 --2- 172.21.15.113:0/980921509 >> v2:172.21.15.141:3300/0 conn(0x7ff760147780 0x7ff76015e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.600+0000 7ff75ffff700 1 -- 172.21.15.113:0/980921509 >> v2:172.21.15.141:3300/0 conn(0x7ff760147780 msgr2=0x7ff76015e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:07.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.600+0000 7ff75ffff700 1 --2- 172.21.15.113:0/980921509 >> v2:172.21.15.141:3300/0 conn(0x7ff760147780 0x7ff76015e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.600+0000 7ff75ffff700 1 -- 172.21.15.113:0/980921509 >> v1:172.21.15.141:6789/0 conn(0x7ff7601473a0 legacy=0x7ff76006c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:07.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.600+0000 7ff75ffff700 1 -- 172.21.15.113:0/980921509 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76015f3a0 con 0x7ff76006dc20 2024-01-15T11:26:07.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.600+0000 7ff75ffff700 1 --2- 172.21.15.113:0/980921509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff76006dc20 0x7ff760073f90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff760147d30 tx=0x7ff75800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d65e74b85097a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:07.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.600+0000 7ff75e7fc700 1 -- 172.21.15.113:0/980921509 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75800e070 con 0x7ff76006dc20 2024-01-15T11:26:07.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.600+0000 7ff75e7fc700 1 -- 172.21.15.113:0/980921509 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff758003a20 con 0x7ff76006dc20 2024-01-15T11:26:07.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.600+0000 7ff75e7fc700 1 -- 172.21.15.113:0/980921509 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff758003dd0 con 0x7ff76006dc20 2024-01-15T11:26:07.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.601+0000 7ff765e91700 1 -- 172.21.15.113:0/980921509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff76006dc20 msgr2=0x7ff760073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.601+0000 7ff765e91700 1 --2- 172.21.15.113:0/980921509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff76006dc20 0x7ff760073f90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff760147d30 tx=0x7ff75800b330 comp rx=0 tx=0).stop 2024-01-15T11:26:07.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.601+0000 7ff765e91700 1 -- 172.21.15.113:0/980921509 shutdown_connections 2024-01-15T11:26:07.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.601+0000 7ff765e91700 1 --2- 172.21.15.113:0/980921509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff76006dc20 0x7ff760073f90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.601+0000 7ff765e91700 1 --2- 172.21.15.113:0/980921509 >> v2:172.21.15.141:3300/0 conn(0x7ff760147780 0x7ff76015e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.601+0000 7ff765e91700 1 -- 172.21.15.113:0/980921509 >> 172.21.15.113:0/980921509 conn(0x7ff7600b3130 msgr2=0x7ff7600d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:07.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.601+0000 7ff765e91700 1 -- 172.21.15.113:0/980921509 shutdown_connections 2024-01-15T11:26:07.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.601+0000 7ff765e91700 1 -- 172.21.15.113:0/980921509 wait complete. 2024-01-15T11:26:07.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.602+0000 7ff765e91700 1 Processor -- start 2024-01-15T11:26:07.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.602+0000 7ff765e91700 1 -- start start 2024-01-15T11:26:07.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff765e91700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff76006dc20 0x7ff7600b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff765e91700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7601473a0 0x7ff7600af420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff765e91700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff760147780 0x7ff7600af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff765e91700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7601e8fd0 con 0x7ff7601473a0 2024-01-15T11:26:07.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff765e91700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7601e8400 con 0x7ff76006dc20 2024-01-15T11:26:07.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff765e91700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7601e86a0 con 0x7ff760147780 2024-01-15T11:26:07.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff75effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7601473a0 0x7ff7600af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff75effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7601473a0 0x7ff7600af420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43498/0 (socket says 172.21.15.113:43498) 2024-01-15T11:26:07.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff75effd700 1 -- 172.21.15.113:0/1120778918 learned_addr learned my addr 172.21.15.113:0/1120778918 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:07.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff75effd700 1 -- 172.21.15.113:0/1120778918 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff760147780 msgr2=0x7ff7600af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff75ffff700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff760147780 0x7ff7600af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff75f7fe700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff76006dc20 0x7ff7600b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff75effd700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff760147780 0x7ff7600af960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff75effd700 1 -- 172.21.15.113:0/1120778918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff76006dc20 msgr2=0x7ff7600b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff75effd700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff76006dc20 0x7ff7600b1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff75effd700 1 -- 172.21.15.113:0/1120778918 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff758009b30 con 0x7ff7601473a0 2024-01-15T11:26:07.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff75f7fe700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff76006dc20 0x7ff7600b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:07.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.603+0000 7ff75effd700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7601473a0 0x7ff7600af420 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff75400d7d0 tx=0x7ff75400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:07.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.604+0000 7ff747fff700 1 -- 172.21.15.113:0/1120778918 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7540125f0 con 0x7ff7601473a0 2024-01-15T11:26:07.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.604+0000 7ff747fff700 1 -- 172.21.15.113:0/1120778918 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff754012750 con 0x7ff7601473a0 2024-01-15T11:26:07.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.604+0000 7ff747fff700 1 -- 172.21.15.113:0/1120778918 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754012ac0 con 0x7ff7601473a0 2024-01-15T11:26:07.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.604+0000 7ff765e91700 1 -- 172.21.15.113:0/1120778918 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7601e8980 con 0x7ff7601473a0 2024-01-15T11:26:07.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.604+0000 7ff765e91700 1 -- 172.21.15.113:0/1120778918 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7600b1950 con 0x7ff7601473a0 2024-01-15T11:26:07.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.604+0000 7ff765e91700 1 -- 172.21.15.113:0/1120778918 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7600b1b10 con 0x7ff7601473a0 2024-01-15T11:26:07.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.605+0000 7ff747fff700 1 -- 172.21.15.113:0/1120778918 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff754012c20 con 0x7ff7601473a0 2024-01-15T11:26:07.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.605+0000 7ff747fff700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff74c06b350 0x7ff74c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.605+0000 7ff747fff700 1 -- 172.21.15.113:0/1120778918 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff754089ba0 con 0x7ff7601473a0 2024-01-15T11:26:07.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.605+0000 7ff75f7fe700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff74c06b350 0x7ff74c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.606+0000 7ff75f7fe700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff74c06b350 0x7ff74c06d810 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff750003f40 tx=0x7ff750009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:07.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.610+0000 7ff747fff700 1 -- 172.21.15.113:0/1120778918 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff754054d60 con 0x7ff7601473a0 2024-01-15T11:26:07.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.800+0000 7ff765e91700 1 -- 172.21.15.113:0/1120778918 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff760067f60 con 0x7ff7601473a0 2024-01-15T11:26:07.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.801+0000 7ff747fff700 1 -- 172.21.15.113:0/1120778918 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+12917 (secure 0 0 0) 0x7ff754012ed0 con 0x7ff7601473a0 2024-01-15T11:26:07.800 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:07.800 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":29,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:26:04.524961+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:26:07.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.803+0000 7ff765e91700 1 -- 172.21.15.113:0/1120778918 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff74c06b350 msgr2=0x7ff74c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.803+0000 7ff765e91700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff74c06b350 0x7ff74c06d810 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff750003f40 tx=0x7ff750009040 comp rx=0 tx=0).stop 2024-01-15T11:26:07.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.803+0000 7ff765e91700 1 -- 172.21.15.113:0/1120778918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7601473a0 msgr2=0x7ff7600af420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.803+0000 7ff765e91700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7601473a0 0x7ff7600af420 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff75400d7d0 tx=0x7ff75400db90 comp rx=0 tx=0).stop 2024-01-15T11:26:07.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.803+0000 7ff765e91700 1 -- 172.21.15.113:0/1120778918 shutdown_connections 2024-01-15T11:26:07.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.803+0000 7ff765e91700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff74c06b350 0x7ff74c06d810 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.803+0000 7ff765e91700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff760147780 0x7ff7600af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.803+0000 7ff765e91700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff76006dc20 0x7ff7600b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.803+0000 7ff765e91700 1 --2- 172.21.15.113:0/1120778918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7601473a0 0x7ff7600af420 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.803+0000 7ff765e91700 1 -- 172.21.15.113:0/1120778918 >> 172.21.15.113:0/1120778918 conn(0x7ff7600b3130 msgr2=0x7ff76006b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:07.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.803+0000 7ff765e91700 1 -- 172.21.15.113:0/1120778918 shutdown_connections 2024-01-15T11:26:07.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.803+0000 7ff765e91700 1 -- 172.21.15.113:0/1120778918 wait complete. 2024-01-15T11:26:07.814 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:26:07.815 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:26:07.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.939+0000 7f2b770ec700 1 Processor -- start 2024-01-15T11:26:07.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.939+0000 7f2b770ec700 1 -- start start 2024-01-15T11:26:07.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.940+0000 7f2b770ec700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7015e110 0x7f2b7006f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.940+0000 7f2b770ec700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7006d980 con 0x7f2b700afb30 2024-01-15T11:26:07.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.940+0000 7f2b770ec700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b700affb0 con 0x7f2b7006e420 2024-01-15T11:26:07.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.940+0000 7f2b770ec700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b700b0260 con 0x7f2b7015e110 2024-01-15T11:26:07.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.940+0000 7f2b6ffff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f2b700afb30 0x7f2b7006aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:43452/0 (socket says 172.21.15.113:43452) 2024-01-15T11:26:07.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.940+0000 7f2b6ffff700 1 -- 172.21.15.113:0/748051421 learned_addr learned my addr 172.21.15.113:0/748051421 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:07.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881705349 0 0) 0x7f2b7006d980 con 0x7f2b700afb30 2024-01-15T11:26:07.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b74e88700 1 --2- 172.21.15.113:0/748051421 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7015e110 0x7f2b7006f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b600035b0 con 0x7f2b700afb30 2024-01-15T11:26:07.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3560827737 0 0) 0x7f2b600035b0 con 0x7f2b700afb30 2024-01-15T11:26:07.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7015e110 msgr2=0x7f2b7006f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:07.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 --2- 172.21.15.113:0/748051421 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7015e110 0x7f2b7006f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 >> v1:172.21.15.141:6789/0 conn(0x7f2b7006e420 legacy=0x7f2b70073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b700b0540 con 0x7f2b700afb30 2024-01-15T11:26:07.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f2b5c001670 con 0x7f2b700afb30 2024-01-15T11:26:07.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 >> v1:172.21.15.113:6789/0 conn(0x7f2b700afb30 legacy=0x7f2b7006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 --2- 172.21.15.113:0/748051421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b60003a30 0x7f2b60003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 --2- 172.21.15.113:0/748051421 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b60004850 0x7f2b60004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 --2- 172.21.15.113:0/748051421 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7015e110 0x7f2b60005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b700b0260 con 0x7f2b60003a30 2024-01-15T11:26:07.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6000baf0 con 0x7f2b60004850 2024-01-15T11:26:07.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6000bcb0 con 0x7f2b7015e110 2024-01-15T11:26:07.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b74e88700 1 --2- 172.21.15.113:0/748051421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b60003a30 0x7f2b60003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b6ffff700 1 --2- 172.21.15.113:0/748051421 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b60004850 0x7f2b60004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b75689700 1 --2- 172.21.15.113:0/748051421 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7015e110 0x7f2b60005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b74e88700 1 -- 172.21.15.113:0/748051421 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7015e110 msgr2=0x7f2b60005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:07.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b74e88700 1 --2- 172.21.15.113:0/748051421 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7015e110 0x7f2b60005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b74e88700 1 -- 172.21.15.113:0/748051421 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b60004850 msgr2=0x7f2b60004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:07.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b74e88700 1 --2- 172.21.15.113:0/748051421 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b60004850 0x7f2b60004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b74e88700 1 -- 172.21.15.113:0/748051421 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b600035b0 con 0x7f2b60003a30 2024-01-15T11:26:07.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.941+0000 7f2b74e88700 1 --2- 172.21.15.113:0/748051421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b60003a30 0x7f2b60003e50 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009b30 tx=0x7f2b60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb0ac2ef32e006d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:07.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.942+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6400e6c0 con 0x7f2b60003a30 2024-01-15T11:26:07.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.942+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6400e820 con 0x7f2b60003a30 2024-01-15T11:26:07.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.942+0000 7f2b6f7fe700 1 -- 172.21.15.113:0/748051421 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6400eb90 con 0x7f2b60003a30 2024-01-15T11:26:07.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.942+0000 7f2b770ec700 1 -- 172.21.15.113:0/748051421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b60003a30 msgr2=0x7f2b60003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.942+0000 7f2b770ec700 1 --2- 172.21.15.113:0/748051421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b60003a30 0x7f2b60003e50 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009b30 tx=0x7f2b60005880 comp rx=0 tx=0).stop 2024-01-15T11:26:07.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.942+0000 7f2b75689700 1 -- 172.21.15.113:0/748051421 reap_dead start 2024-01-15T11:26:07.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.942+0000 7f2b770ec700 1 -- 172.21.15.113:0/748051421 shutdown_connections 2024-01-15T11:26:07.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.942+0000 7f2b770ec700 1 -- 172.21.15.113:0/748051421 >> 172.21.15.113:0/748051421 conn(0x7f2b700b3250 msgr2=0x7f2b700d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:07.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.942+0000 7f2b770ec700 1 -- 172.21.15.113:0/748051421 shutdown_connections 2024-01-15T11:26:07.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.942+0000 7f2b770ec700 1 -- 172.21.15.113:0/748051421 wait complete. 2024-01-15T11:26:07.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.943+0000 7f2b770ec700 1 Processor -- start 2024-01-15T11:26:07.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.943+0000 7f2b770ec700 1 -- start start 2024-01-15T11:26:07.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.944+0000 7f2b770ec700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b60003a30 0x7f2b70148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.944+0000 7f2b770ec700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7014b490 0x7f2b70149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.944+0000 7f2b770ec700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b70149e70 0x7f2b700b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.944+0000 7f2b770ec700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7014b070 con 0x7f2b60003a30 2024-01-15T11:26:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.944+0000 7f2b770ec700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7014a3f0 con 0x7f2b70149e70 2024-01-15T11:26:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.944+0000 7f2b770ec700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7014a6d0 con 0x7f2b7014b490 2024-01-15T11:26:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.944+0000 7f2b74e88700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b60003a30 0x7f2b70148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.944+0000 7f2b74e88700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b60003a30 0x7f2b70148fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43522/0 (socket says 172.21.15.113:43522) 2024-01-15T11:26:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.944+0000 7f2b74e88700 1 -- 172.21.15.113:0/3016533641 learned_addr learned my addr 172.21.15.113:0/3016533641 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b75689700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b70149e70 0x7f2b700b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b6ffff700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7014b490 0x7f2b70149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b74e88700 1 -- 172.21.15.113:0/3016533641 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7014b490 msgr2=0x7f2b70149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b74e88700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7014b490 0x7f2b70149930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b74e88700 1 -- 172.21.15.113:0/3016533641 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b70149e70 msgr2=0x7f2b700b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b74e88700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b70149e70 0x7f2b700b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b74e88700 1 -- 172.21.15.113:0/3016533641 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7006b880 con 0x7f2b60003a30 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b74e88700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b60003a30 0x7f2b70148fa0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000bac0 tx=0x7f2b640047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b6d7fa700 1 -- 172.21.15.113:0/3016533641 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6400e6c0 con 0x7f2b60003a30 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b6d7fa700 1 -- 172.21.15.113:0/3016533641 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b64003ad0 con 0x7f2b60003a30 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b770ec700 1 -- 172.21.15.113:0/3016533641 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7014a9b0 con 0x7f2b60003a30 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b6d7fa700 1 -- 172.21.15.113:0/3016533641 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64003e80 con 0x7f2b60003a30 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.945+0000 7f2b770ec700 1 -- 172.21.15.113:0/3016533641 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b70158bf0 con 0x7f2b60003a30 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.946+0000 7f2b770ec700 1 -- 172.21.15.113:0/3016533641 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b70069020 con 0x7f2b60003a30 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.946+0000 7f2b6d7fa700 1 -- 172.21.15.113:0/3016533641 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2b64013470 con 0x7f2b60003a30 2024-01-15T11:26:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.947+0000 7f2b6d7fa700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2b5406b350 0x7f2b5406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.947+0000 7f2b6d7fa700 1 -- 172.21.15.113:0/3016533641 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2b6408b0f0 con 0x7f2b60003a30 2024-01-15T11:26:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.947+0000 7f2b6ffff700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2b5406b350 0x7f2b5406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.947+0000 7f2b6ffff700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2b5406b350 0x7f2b5406d810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c001eb0 tx=0x7f2b5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:07.951+0000 7f2b6d7fa700 1 -- 172.21.15.113:0/3016533641 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b64056460 con 0x7f2b60003a30 2024-01-15T11:26:08.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.139+0000 7f2b770ec700 1 -- 172.21.15.113:0/3016533641 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2b700b1cb0 con 0x7f2b60003a30 2024-01-15T11:26:08.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.140+0000 7f2b6d7fa700 1 -- 172.21.15.113:0/3016533641 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2b64059a80 con 0x7f2b60003a30 2024-01-15T11:26:08.139 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:26:08.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.142+0000 7f2b770ec700 1 -- 172.21.15.113:0/3016533641 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2b5406b350 msgr2=0x7f2b5406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.142+0000 7f2b770ec700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2b5406b350 0x7f2b5406d810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c001eb0 tx=0x7f2b5c009040 comp rx=0 tx=0).stop 2024-01-15T11:26:08.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.142+0000 7f2b770ec700 1 -- 172.21.15.113:0/3016533641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b60003a30 msgr2=0x7f2b70148fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.142+0000 7f2b770ec700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b60003a30 0x7f2b70148fa0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000bac0 tx=0x7f2b640047c0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.142+0000 7f2b770ec700 1 -- 172.21.15.113:0/3016533641 shutdown_connections 2024-01-15T11:26:08.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.142+0000 7f2b770ec700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2b5406b350 0x7f2b5406d810 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.142+0000 7f2b770ec700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2b70149e70 0x7f2b700b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.142+0000 7f2b770ec700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b60003a30 0x7f2b70148fa0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.142+0000 7f2b770ec700 1 --2- 172.21.15.113:0/3016533641 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2b7014b490 0x7f2b70149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.142+0000 7f2b770ec700 1 -- 172.21.15.113:0/3016533641 >> 172.21.15.113:0/3016533641 conn(0x7f2b700b3250 msgr2=0x7f2b700d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:08.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.142+0000 7f2b770ec700 1 -- 172.21.15.113:0/3016533641 shutdown_connections 2024-01-15T11:26:08.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.142+0000 7f2b770ec700 1 -- 172.21.15.113:0/3016533641 wait complete. 2024-01-15T11:26:08.153 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T11:26:08.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.275+0000 7f0720e09700 1 Processor -- start 2024-01-15T11:26:08.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.275+0000 7f0720e09700 1 -- start start 2024-01-15T11:26:08.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f0720e09700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f071c15e280 0x7f071c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:08.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f0720e09700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f071c0cf250 0x7f071c06b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:08.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f0720e09700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f071c0d0a80 con 0x7f071c15dea0 2024-01-15T11:26:08.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f0720e09700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f071c0d2670 con 0x7f071c0cf250 2024-01-15T11:26:08.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f0720e09700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071c06c4a0 con 0x7f071c15e280 2024-01-15T11:26:08.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f071ad9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f071c0cf250 0x7f071c06b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:08.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f071a59c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f071c15dea0 0x7f071c0d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:43458/0 (socket says 172.21.15.113:43458) 2024-01-15T11:26:08.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f071a59c700 1 -- 172.21.15.113:0/2956024306 learned_addr learned my addr 172.21.15.113:0/2956024306 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:08.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f0719d9b700 1 --2- 172.21.15.113:0/2956024306 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f071c15e280 0x7f071c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:08.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f071ad9d700 1 -- 172.21.15.113:0/2956024306 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f071c15e280 msgr2=0x7f071c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:08.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f071ad9d700 1 --2- 172.21.15.113:0/2956024306 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f071c15e280 0x7f071c0cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f071ad9d700 1 -- 172.21.15.113:0/2956024306 >> v1:172.21.15.113:6789/0 conn(0x7f071c15dea0 legacy=0x7f071c0d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f071ad9d700 1 -- 172.21.15.113:0/2956024306 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071c0cfeb0 con 0x7f071c0cf250 2024-01-15T11:26:08.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f071ad9d700 1 --2- 172.21.15.113:0/2956024306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f071c0cf250 0x7f071c06b8e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0714009d90 tx=0x7f071400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=278c471936ef0417 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:08.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.276+0000 7f071959a700 1 -- 172.21.15.113:0/2956024306 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071400e070 con 0x7f071c0cf250 2024-01-15T11:26:08.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.277+0000 7f071959a700 1 -- 172.21.15.113:0/2956024306 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071400bbd0 con 0x7f071c0cf250 2024-01-15T11:26:08.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.277+0000 7f071959a700 1 -- 172.21.15.113:0/2956024306 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0714012430 con 0x7f071c0cf250 2024-01-15T11:26:08.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.277+0000 7f0720e09700 1 -- 172.21.15.113:0/2956024306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f071c0cf250 msgr2=0x7f071c06b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.277+0000 7f0720e09700 1 --2- 172.21.15.113:0/2956024306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f071c0cf250 0x7f071c06b8e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0714009d90 tx=0x7f071400b330 comp rx=0 tx=0).stop 2024-01-15T11:26:08.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.277+0000 7f0720e09700 1 -- 172.21.15.113:0/2956024306 shutdown_connections 2024-01-15T11:26:08.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.277+0000 7f0720e09700 1 --2- 172.21.15.113:0/2956024306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f071c0cf250 0x7f071c06b8e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.277+0000 7f0720e09700 1 --2- 172.21.15.113:0/2956024306 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f071c15e280 0x7f071c0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.277+0000 7f0720e09700 1 -- 172.21.15.113:0/2956024306 >> 172.21.15.113:0/2956024306 conn(0x7f071c073130 msgr2=0x7f071c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:08.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.277+0000 7f0720e09700 1 -- 172.21.15.113:0/2956024306 shutdown_connections 2024-01-15T11:26:08.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.277+0000 7f0720e09700 1 -- 172.21.15.113:0/2956024306 wait complete. 2024-01-15T11:26:08.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.278+0000 7f0720e09700 1 Processor -- start 2024-01-15T11:26:08.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.278+0000 7f0720e09700 1 -- start start 2024-01-15T11:26:08.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.279+0000 7f0720e09700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f071c0cf250 0x7f071c0b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:08.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.279+0000 7f0720e09700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f071c15dea0 0x7f071c1f3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:08.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.279+0000 7f0720e09700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f071c0cd350 0x7f071c15a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:08.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.279+0000 7f0720e09700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c0cdb30 con 0x7f071c15dea0 2024-01-15T11:26:08.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.279+0000 7f0720e09700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c0cd850 con 0x7f071c0cd350 2024-01-15T11:26:08.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.279+0000 7f0720e09700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071c1596e0 con 0x7f071c0cf250 2024-01-15T11:26:08.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.279+0000 7f0719d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f071c15dea0 0x7f071c1f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:08.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.279+0000 7f0719d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f071c15dea0 0x7f071c1f3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43548/0 (socket says 172.21.15.113:43548) 2024-01-15T11:26:08.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.279+0000 7f0719d9b700 1 -- 172.21.15.113:0/3522450534 learned_addr learned my addr 172.21.15.113:0/3522450534 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:08.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.279+0000 7f0719d9b700 1 -- 172.21.15.113:0/3522450534 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f071c0cf250 msgr2=0x7f071c0b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.279+0000 7f0719d9b700 1 --2- 172.21.15.113:0/3522450534 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f071c0cf250 0x7f071c0b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.280+0000 7f0719d9b700 1 -- 172.21.15.113:0/3522450534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f071c0cd350 msgr2=0x7f071c15a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.280+0000 7f0719d9b700 1 --2- 172.21.15.113:0/3522450534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f071c0cd350 0x7f071c15a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.280+0000 7f0719d9b700 1 -- 172.21.15.113:0/3522450534 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0714009a70 con 0x7f071c15dea0 2024-01-15T11:26:08.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.280+0000 7f0719d9b700 1 --2- 172.21.15.113:0/3522450534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f071c15dea0 0x7f071c1f3150 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f070c00c7f0 tx=0x7f070c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:08.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.280+0000 7f070affd700 1 -- 172.21.15.113:0/3522450534 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c011800 con 0x7f071c15dea0 2024-01-15T11:26:08.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.280+0000 7f070affd700 1 -- 172.21.15.113:0/3522450534 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070c011960 con 0x7f071c15dea0 2024-01-15T11:26:08.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.280+0000 7f0720e09700 1 -- 172.21.15.113:0/3522450534 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071c1599c0 con 0x7f071c15dea0 2024-01-15T11:26:08.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.280+0000 7f070affd700 1 -- 172.21.15.113:0/3522450534 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c011cf0 con 0x7f071c15dea0 2024-01-15T11:26:08.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.280+0000 7f0720e09700 1 -- 172.21.15.113:0/3522450534 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071c0b0d40 con 0x7f071c15dea0 2024-01-15T11:26:08.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.281+0000 7f0720e09700 1 -- 172.21.15.113:0/3522450534 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071c067f60 con 0x7f071c15dea0 2024-01-15T11:26:08.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.281+0000 7f070affd700 1 -- 172.21.15.113:0/3522450534 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f070c018360 con 0x7f071c15dea0 2024-01-15T11:26:08.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.281+0000 7f070affd700 1 --2- 172.21.15.113:0/3522450534 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f06fc06b350 0x7f06fc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:08.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.282+0000 7f070affd700 1 -- 172.21.15.113:0/3522450534 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f070c089760 con 0x7f071c15dea0 2024-01-15T11:26:08.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.282+0000 7f071a59c700 1 --2- 172.21.15.113:0/3522450534 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f06fc06b350 0x7f06fc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:08.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.282+0000 7f071a59c700 1 --2- 172.21.15.113:0/3522450534 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f06fc06b350 0x7f06fc06d810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0710001670 tx=0x7f0710007ad0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:08.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.286+0000 7f070affd700 1 -- 172.21.15.113:0/3522450534 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f070c054920 con 0x7f071c15dea0 2024-01-15T11:26:08.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.479+0000 7f0720e09700 1 -- 172.21.15.113:0/3522450534 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f071c0b7bf0 con 0x7f071c15dea0 2024-01-15T11:26:08.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.480+0000 7f070affd700 1 -- 172.21.15.113:0/3522450534 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) v1 ==== 84+0+65 (secure 0 0 0) 0x7f070c01f060 con 0x7f071c15dea0 2024-01-15T11:26:08.479 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:08.479 INFO:teuthology.orchestra.run.smithi113.stdout:[ 2024-01-15T11:26:08.479 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:08.479 INFO:teuthology.orchestra.run.smithi113.stdout: "poolnum": 1, 2024-01-15T11:26:08.479 INFO:teuthology.orchestra.run.smithi113.stdout: "poolname": ".mgr" 2024-01-15T11:26:08.480 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:08.480 INFO:teuthology.orchestra.run.smithi113.stdout:] 2024-01-15T11:26:08.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.482+0000 7f0720e09700 1 -- 172.21.15.113:0/3522450534 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f06fc06b350 msgr2=0x7f06fc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.482+0000 7f0720e09700 1 --2- 172.21.15.113:0/3522450534 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f06fc06b350 0x7f06fc06d810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0710001670 tx=0x7f0710007ad0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.482+0000 7f0720e09700 1 -- 172.21.15.113:0/3522450534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f071c15dea0 msgr2=0x7f071c1f3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.482+0000 7f0720e09700 1 --2- 172.21.15.113:0/3522450534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f071c15dea0 0x7f071c1f3150 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f070c00c7f0 tx=0x7f070c00cbb0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.482+0000 7f0720e09700 1 -- 172.21.15.113:0/3522450534 shutdown_connections 2024-01-15T11:26:08.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.482+0000 7f0720e09700 1 --2- 172.21.15.113:0/3522450534 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f06fc06b350 0x7f06fc06d810 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.482+0000 7f0720e09700 1 --2- 172.21.15.113:0/3522450534 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f071c0cd350 0x7f071c15a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.482+0000 7f0720e09700 1 --2- 172.21.15.113:0/3522450534 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f071c0cf250 0x7f071c0b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.482+0000 7f0720e09700 1 --2- 172.21.15.113:0/3522450534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f071c15dea0 0x7f071c1f3150 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.482+0000 7f0720e09700 1 -- 172.21.15.113:0/3522450534 >> 172.21.15.113:0/3522450534 conn(0x7f071c073130 msgr2=0x7f071c0d0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:08.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.482+0000 7f0720e09700 1 -- 172.21.15.113:0/3522450534 shutdown_connections 2024-01-15T11:26:08.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.482+0000 7f0720e09700 1 -- 172.21.15.113:0/3522450534 wait complete. 2024-01-15T11:26:08.493 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T11:26:08.493 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T11:26:08.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.621+0000 7f2dce250700 1 Processor -- start 2024-01-15T11:26:08.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.621+0000 7f2dce250700 1 -- start start 2024-01-15T11:26:08.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dce250700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2dc815a4e0 0x7f2dc80726c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:08.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dce250700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc80ce160 con 0x7f2dc80b1890 2024-01-15T11:26:08.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dce250700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc80cdc20 con 0x7f2dc80af730 2024-01-15T11:26:08.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dce250700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc80ce970 con 0x7f2dc815a4e0 2024-01-15T11:26:08.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dc77fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2dc815a4e0 0x7f2dc80726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:08.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dc6ffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f2dc80b1890 0x7f2dc8072c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:43472/0 (socket says 172.21.15.113:43472) 2024-01-15T11:26:08.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dc6ffd700 1 -- 172.21.15.113:0/2586653455 learned_addr learned my addr 172.21.15.113:0/2586653455 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:08.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dc77fe700 1 -- 172.21.15.113:0/2586653455 >> v1:172.21.15.141:6789/0 conn(0x7f2dc80af730 legacy=0x7f2dc80afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dc77fe700 1 -- 172.21.15.113:0/2586653455 >> v1:172.21.15.113:6789/0 conn(0x7f2dc80b1890 legacy=0x7f2dc8072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dc77fe700 1 -- 172.21.15.113:0/2586653455 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc80ce3b0 con 0x7f2dc815a4e0 2024-01-15T11:26:08.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dc67fc700 1 -- 172.21.15.113:0/2586653455 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48783363 0 0) 0x7f2dc80ce160 con 0x7f2dc80b1890 2024-01-15T11:26:08.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dc77fe700 1 --2- 172.21.15.113:0/2586653455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dc815a4e0 0x7f2dc80726c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009840 tx=0x7f2db8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac64bf74631ab333 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:08.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dc67fc700 1 -- 172.21.15.113:0/2586653455 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8011830 con 0x7f2dc815a4e0 2024-01-15T11:26:08.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dc67fc700 1 -- 172.21.15.113:0/2586653455 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db8011990 con 0x7f2dc815a4e0 2024-01-15T11:26:08.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.622+0000 7f2dc67fc700 1 -- 172.21.15.113:0/2586653455 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8011d20 con 0x7f2dc815a4e0 2024-01-15T11:26:08.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.623+0000 7f2dce250700 1 -- 172.21.15.113:0/2586653455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dc815a4e0 msgr2=0x7f2dc80726c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.623+0000 7f2dce250700 1 --2- 172.21.15.113:0/2586653455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dc815a4e0 0x7f2dc80726c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009840 tx=0x7f2db8009b50 comp rx=0 tx=0).stop 2024-01-15T11:26:08.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.623+0000 7f2dce250700 1 -- 172.21.15.113:0/2586653455 shutdown_connections 2024-01-15T11:26:08.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.623+0000 7f2dce250700 1 --2- 172.21.15.113:0/2586653455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dc815a4e0 0x7f2dc80726c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.623+0000 7f2dce250700 1 -- 172.21.15.113:0/2586653455 >> 172.21.15.113:0/2586653455 conn(0x7f2dc80b1e90 msgr2=0x7f2dc80b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:08.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.623+0000 7f2dce250700 1 -- 172.21.15.113:0/2586653455 shutdown_connections 2024-01-15T11:26:08.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.623+0000 7f2dce250700 1 -- 172.21.15.113:0/2586653455 wait complete. 2024-01-15T11:26:08.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.624+0000 7f2dce250700 1 Processor -- start 2024-01-15T11:26:08.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.624+0000 7f2dce250700 1 -- start start 2024-01-15T11:26:08.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dce250700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2dc80af730 0x7f2dc809dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:08.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dce250700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dc815a4e0 0x7f2dc8149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:08.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dce250700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2dc814c720 0x7f2dc8149b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:08.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dce250700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc809ae70 con 0x7f2dc815a4e0 2024-01-15T11:26:08.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dce250700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc814ce90 con 0x7f2dc80af730 2024-01-15T11:26:08.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dce250700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc809a280 con 0x7f2dc814c720 2024-01-15T11:26:08.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dc6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dc815a4e0 0x7f2dc8149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:08.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dc6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dc815a4e0 0x7f2dc8149640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43558/0 (socket says 172.21.15.113:43558) 2024-01-15T11:26:08.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dc6ffd700 1 -- 172.21.15.113:0/1586000328 learned_addr learned my addr 172.21.15.113:0/1586000328 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:08.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dc6ffd700 1 -- 172.21.15.113:0/1586000328 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2dc814c720 msgr2=0x7f2dc8149b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dc6ffd700 1 --2- 172.21.15.113:0/1586000328 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2dc814c720 0x7f2dc8149b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dc6ffd700 1 -- 172.21.15.113:0/1586000328 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2dc80af730 msgr2=0x7f2dc809dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:08.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dc6ffd700 1 --2- 172.21.15.113:0/1586000328 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2dc80af730 0x7f2dc809dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:08.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dc6ffd700 1 -- 172.21.15.113:0/1586000328 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db80094a0 con 0x7f2dc815a4e0 2024-01-15T11:26:08.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2dc6ffd700 1 --2- 172.21.15.113:0/1586000328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dc815a4e0 0x7f2dc8149640 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00c520 tx=0x7f2dbc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:08.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2daffff700 1 -- 172.21.15.113:0/1586000328 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc004c80 con 0x7f2dc815a4e0 2024-01-15T11:26:08.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.625+0000 7f2daffff700 1 -- 172.21.15.113:0/1586000328 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dbc004e00 con 0x7f2dc815a4e0 2024-01-15T11:26:08.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.626+0000 7f2dce250700 1 -- 172.21.15.113:0/1586000328 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc814a1b0 con 0x7f2dc815a4e0 2024-01-15T11:26:08.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.626+0000 7f2daffff700 1 -- 172.21.15.113:0/1586000328 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc007310 con 0x7f2dc815a4e0 2024-01-15T11:26:08.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.626+0000 7f2dce250700 1 -- 172.21.15.113:0/1586000328 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc809a7b0 con 0x7f2dc815a4e0 2024-01-15T11:26:08.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.626+0000 7f2dce250700 1 -- 172.21.15.113:0/1586000328 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc809afb0 con 0x7f2dc815a4e0 2024-01-15T11:26:08.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.627+0000 7f2daffff700 1 -- 172.21.15.113:0/1586000328 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2dbc007470 con 0x7f2dc815a4e0 2024-01-15T11:26:08.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.628+0000 7f2daffff700 1 --2- 172.21.15.113:0/1586000328 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2db406b350 0x7f2db406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:08.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.628+0000 7f2daffff700 1 -- 172.21.15.113:0/1586000328 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2dbc089a20 con 0x7f2dc815a4e0 2024-01-15T11:26:08.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.628+0000 7f2dc77fe700 1 --2- 172.21.15.113:0/1586000328 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2db406b350 0x7f2db406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:08.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.628+0000 7f2dc77fe700 1 --2- 172.21.15.113:0/1586000328 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2db406b350 0x7f2db406d810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2db80090b0 tx=0x7f2db8010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:08.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.632+0000 7f2daffff700 1 -- 172.21.15.113:0/1586000328 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2dbc054c50 con 0x7f2dc815a4e0 2024-01-15T11:26:08.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:08.821+0000 7f2dce250700 1 -- 172.21.15.113:0/1586000328 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2dc809afb0 con 0x7f2dc815a4e0 2024-01-15T11:26:09.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.530+0000 7f2daffff700 1 -- 172.21.15.113:0/1586000328 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2dc809afb0 con 0x7f2dc815a4e0 2024-01-15T11:26:09.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.534+0000 7f2dce250700 1 -- 172.21.15.113:0/1586000328 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2db406b350 msgr2=0x7f2db406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.534+0000 7f2dce250700 1 --2- 172.21.15.113:0/1586000328 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2db406b350 0x7f2db406d810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2db80090b0 tx=0x7f2db8010000 comp rx=0 tx=0).stop 2024-01-15T11:26:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.534+0000 7f2dce250700 1 -- 172.21.15.113:0/1586000328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dc815a4e0 msgr2=0x7f2dc8149640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.534+0000 7f2dce250700 1 --2- 172.21.15.113:0/1586000328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dc815a4e0 0x7f2dc8149640 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00c520 tx=0x7f2dbc00c8e0 comp rx=0 tx=0).stop 2024-01-15T11:26:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.534+0000 7f2dce250700 1 -- 172.21.15.113:0/1586000328 shutdown_connections 2024-01-15T11:26:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.534+0000 7f2dce250700 1 --2- 172.21.15.113:0/1586000328 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2db406b350 0x7f2db406d810 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.534+0000 7f2dce250700 1 --2- 172.21.15.113:0/1586000328 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2dc814c720 0x7f2dc8149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.534+0000 7f2dce250700 1 --2- 172.21.15.113:0/1586000328 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2dc80af730 0x7f2dc809dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.534+0000 7f2dce250700 1 --2- 172.21.15.113:0/1586000328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dc815a4e0 0x7f2dc8149640 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:09.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.534+0000 7f2dce250700 1 -- 172.21.15.113:0/1586000328 >> 172.21.15.113:0/1586000328 conn(0x7f2dc80b1e90 msgr2=0x7f2dc806def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:09.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.534+0000 7f2dce250700 1 -- 172.21.15.113:0/1586000328 shutdown_connections 2024-01-15T11:26:09.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.534+0000 7f2dce250700 1 -- 172.21.15.113:0/1586000328 wait complete. 2024-01-15T11:26:09.535 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'cephfs_metadata' created 2024-01-15T11:26:09.550 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T11:26:09.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.684+0000 7f3ebcb0b700 1 Processor -- start 2024-01-15T11:26:09.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.684+0000 7f3ebcb0b700 1 -- start start 2024-01-15T11:26:09.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.685+0000 7f3ebcb0b700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f3eb80afb80 0x7f3eb8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:09.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.685+0000 7f3ebcb0b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3eb80744d0 0x7f3eb80add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:09.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.685+0000 7f3ebcb0b700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb815e4a0 con 0x7f3eb815e0c0 2024-01-15T11:26:09.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.685+0000 7f3ebcb0b700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb80b0010 con 0x7f3eb80744d0 2024-01-15T11:26:09.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.685+0000 7f3ebcb0b700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb8074940 con 0x7f3eb80afb80 2024-01-15T11:26:09.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.685+0000 7f3eb6d9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3eb80744d0 0x7f3eb80add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:09.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.685+0000 7f3eb6d9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3eb80744d0 0x7f3eb80add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43570/0 (socket says 172.21.15.113:43570) 2024-01-15T11:26:09.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.685+0000 7f3eb6d9d700 1 -- 172.21.15.113:0/420385523 learned_addr learned my addr 172.21.15.113:0/420385523 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:09.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.686+0000 7f3eb5d9b700 1 --2- 172.21.15.113:0/420385523 >> v2:172.21.15.141:3300/0 conn(0x7f3eb80afb80 0x7f3eb8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:09.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.686+0000 7f3eb6d9d700 1 -- 172.21.15.113:0/420385523 >> v2:172.21.15.141:3300/0 conn(0x7f3eb80afb80 msgr2=0x7f3eb8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:09.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.686+0000 7f3eb6d9d700 1 --2- 172.21.15.113:0/420385523 >> v2:172.21.15.141:3300/0 conn(0x7f3eb80afb80 0x7f3eb8073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:09.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.686+0000 7f3eb6d9d700 1 -- 172.21.15.113:0/420385523 >> v1:172.21.15.141:6789/0 conn(0x7f3eb815e0c0 legacy=0x7f3eb80b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:09.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.686+0000 7f3eb6d9d700 1 -- 172.21.15.113:0/420385523 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb806aff0 con 0x7f3eb80744d0 2024-01-15T11:26:09.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.686+0000 7f3eb6d9d700 1 --2- 172.21.15.113:0/420385523 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb80744d0 0x7f3eb80add40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0009e80 tx=0x7f3eb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25765c56e5779e3a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:09.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.686+0000 7f3eb559a700 1 -- 172.21.15.113:0/420385523 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb000e070 con 0x7f3eb80744d0 2024-01-15T11:26:09.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.686+0000 7f3eb559a700 1 -- 172.21.15.113:0/420385523 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb000b940 con 0x7f3eb80744d0 2024-01-15T11:26:09.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.686+0000 7f3eb559a700 1 -- 172.21.15.113:0/420385523 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb000bcf0 con 0x7f3eb80744d0 2024-01-15T11:26:09.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.687+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/420385523 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb80744d0 msgr2=0x7f3eb80add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:09.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.687+0000 7f3ebcb0b700 1 --2- 172.21.15.113:0/420385523 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb80744d0 0x7f3eb80add40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0009e80 tx=0x7f3eb000b330 comp rx=0 tx=0).stop 2024-01-15T11:26:09.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.687+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/420385523 shutdown_connections 2024-01-15T11:26:09.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.687+0000 7f3ebcb0b700 1 --2- 172.21.15.113:0/420385523 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb80744d0 0x7f3eb80add40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:09.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.687+0000 7f3ebcb0b700 1 --2- 172.21.15.113:0/420385523 >> v2:172.21.15.141:3300/0 conn(0x7f3eb80afb80 0x7f3eb8073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:09.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.687+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/420385523 >> 172.21.15.113:0/420385523 conn(0x7f3eb80d0c00 msgr2=0x7f3eb80d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:09.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.687+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/420385523 shutdown_connections 2024-01-15T11:26:09.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.687+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/420385523 wait complete. 2024-01-15T11:26:09.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.688+0000 7f3ebcb0b700 1 Processor -- start 2024-01-15T11:26:09.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.688+0000 7f3ebcb0b700 1 -- start start 2024-01-15T11:26:09.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3ebcb0b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3eb80744d0 0x7f3eb81596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:09.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3ebcb0b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb80afb80 0x7f3eb80cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:09.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3ebcb0b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3eb815e0c0 0x7f3eb8147680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:09.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3ebcb0b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb8147bc0 con 0x7f3eb80afb80 2024-01-15T11:26:09.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3ebcb0b700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb8158630 con 0x7f3eb815e0c0 2024-01-15T11:26:09.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3ebcb0b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb8158910 con 0x7f3eb80744d0 2024-01-15T11:26:09.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3eb5d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb80afb80 0x7f3eb80cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:09.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3eb5d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb80afb80 0x7f3eb80cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43580/0 (socket says 172.21.15.113:43580) 2024-01-15T11:26:09.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3eb5d9b700 1 -- 172.21.15.113:0/2188093892 learned_addr learned my addr 172.21.15.113:0/2188093892 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:09.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3eb5d9b700 1 -- 172.21.15.113:0/2188093892 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3eb80744d0 msgr2=0x7f3eb81596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:09.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3eb5d9b700 1 --2- 172.21.15.113:0/2188093892 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3eb80744d0 0x7f3eb81596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:09.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3eb5d9b700 1 -- 172.21.15.113:0/2188093892 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3eb815e0c0 msgr2=0x7f3eb8147680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:09.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3eb5d9b700 1 --2- 172.21.15.113:0/2188093892 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3eb815e0c0 0x7f3eb8147680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:09.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3eb5d9b700 1 -- 172.21.15.113:0/2188093892 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb0009b30 con 0x7f3eb80afb80 2024-01-15T11:26:09.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.689+0000 7f3eb5d9b700 1 --2- 172.21.15.113:0/2188093892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb80afb80 0x7f3eb80cd170 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800e810 tx=0x7f3ea800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:09.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.690+0000 7f3eaeffd700 1 -- 172.21.15.113:0/2188093892 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8013610 con 0x7f3eb80afb80 2024-01-15T11:26:09.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.690+0000 7f3eaeffd700 1 -- 172.21.15.113:0/2188093892 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea8013770 con 0x7f3eb80afb80 2024-01-15T11:26:09.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.690+0000 7f3eaeffd700 1 -- 172.21.15.113:0/2188093892 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8013b20 con 0x7f3eb80afb80 2024-01-15T11:26:09.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.690+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/2188093892 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb8158bf0 con 0x7f3eb80afb80 2024-01-15T11:26:09.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.690+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/2188093892 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb81591b0 con 0x7f3eb80afb80 2024-01-15T11:26:09.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.691+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/2188093892 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb8069020 con 0x7f3eb80afb80 2024-01-15T11:26:09.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.691+0000 7f3eaeffd700 1 -- 172.21.15.113:0/2188093892 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3ea8013c80 con 0x7f3eb80afb80 2024-01-15T11:26:09.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.692+0000 7f3eaeffd700 1 --2- 172.21.15.113:0/2188093892 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3e9806b350 0x7f3e9806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:09.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.692+0000 7f3eaeffd700 1 -- 172.21.15.113:0/2188093892 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f3ea8089690 con 0x7f3eb80afb80 2024-01-15T11:26:09.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.692+0000 7f3eb659c700 1 --2- 172.21.15.113:0/2188093892 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3e9806b350 0x7f3e9806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:09.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.692+0000 7f3eb659c700 1 --2- 172.21.15.113:0/2188093892 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3e9806b350 0x7f3e9806d810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0000f60 tx=0x7f3ea0007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:09.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.696+0000 7f3eaeffd700 1 -- 172.21.15.113:0/2188093892 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ea8054770 con 0x7f3eb80afb80 2024-01-15T11:26:09.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:09.889+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/2188093892 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3eb80d18f0 con 0x7f3eb80afb80 2024-01-15T11:26:10.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.534+0000 7f3eaeffd700 1 -- 172.21.15.113:0/2188093892 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3ea8057d90 con 0x7f3eb80afb80 2024-01-15T11:26:10.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.538+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/2188093892 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3e9806b350 msgr2=0x7f3e9806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:10.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.538+0000 7f3ebcb0b700 1 --2- 172.21.15.113:0/2188093892 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3e9806b350 0x7f3e9806d810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0000f60 tx=0x7f3ea0007a40 comp rx=0 tx=0).stop 2024-01-15T11:26:10.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.538+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/2188093892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb80afb80 msgr2=0x7f3eb80cd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:10.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.538+0000 7f3ebcb0b700 1 --2- 172.21.15.113:0/2188093892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb80afb80 0x7f3eb80cd170 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800e810 tx=0x7f3ea800ebd0 comp rx=0 tx=0).stop 2024-01-15T11:26:10.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.539+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/2188093892 shutdown_connections 2024-01-15T11:26:10.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.539+0000 7f3ebcb0b700 1 --2- 172.21.15.113:0/2188093892 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3e9806b350 0x7f3e9806d810 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:10.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.539+0000 7f3ebcb0b700 1 --2- 172.21.15.113:0/2188093892 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3eb815e0c0 0x7f3eb8147680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:10.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.539+0000 7f3ebcb0b700 1 --2- 172.21.15.113:0/2188093892 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3eb80744d0 0x7f3eb81596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:10.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.539+0000 7f3ebcb0b700 1 --2- 172.21.15.113:0/2188093892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb80afb80 0x7f3eb80cd170 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:10.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.539+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/2188093892 >> 172.21.15.113:0/2188093892 conn(0x7f3eb80d0c00 msgr2=0x7f3eb8151150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:10.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.539+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/2188093892 shutdown_connections 2024-01-15T11:26:10.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.539+0000 7f3ebcb0b700 1 -- 172.21.15.113:0/2188093892 wait complete. 2024-01-15T11:26:10.542 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'cephfs_data' created 2024-01-15T11:26:10.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T11:26:10.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.692+0000 7f728adfa700 1 Processor -- start 2024-01-15T11:26:10.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.692+0000 7f728adfa700 1 -- start start 2024-01-15T11:26:10.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f728adfa700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7284161690 0x7f72840d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:10.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f728adfa700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7284161a70 0x7f72840d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:10.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f728adfa700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72840d0a70 con 0x7f72840d1730 2024-01-15T11:26:10.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f728adfa700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72840d1c10 con 0x7f7284161690 2024-01-15T11:26:10.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f728adfa700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72840b2ab0 con 0x7f7284161a70 2024-01-15T11:26:10.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f7289397700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f72840d1730 0x7f728409aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:43480/0 (socket says 172.21.15.113:43480) 2024-01-15T11:26:10.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f7289397700 1 -- 172.21.15.113:0/1876766963 learned_addr learned my addr 172.21.15.113:0/1876766963 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:10.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363221439 0 0) 0x7f72840d0a70 con 0x7f72840d1730 2024-01-15T11:26:10.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72740035b0 con 0x7f72840d1730 2024-01-15T11:26:10.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f7283fff700 1 --2- 172.21.15.113:0/1876766963 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7284161a70 0x7f72840d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f7288b96700 1 --2- 172.21.15.113:0/1876766963 >> v2:172.21.15.141:3300/0 conn(0x7f7284161690 0x7f72840d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1034893289 0 0) 0x7f72740035b0 con 0x7f72840d1730 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7284161a70 msgr2=0x7f72840d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f72837fe700 1 --2- 172.21.15.113:0/1876766963 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7284161a70 0x7f72840d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 >> v2:172.21.15.141:3300/0 conn(0x7f7284161690 msgr2=0x7f72840d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f72837fe700 1 --2- 172.21.15.113:0/1876766963 >> v2:172.21.15.141:3300/0 conn(0x7f7284161690 0x7f72840d0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.693+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72840b2d90 con 0x7f72840d1730 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f7288b96700 1 --2- 172.21.15.113:0/1876766963 >> v2:172.21.15.141:3300/0 conn(0x7f7284161690 0x7f72840d0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f727c0019f0 con 0x7f72840d1730 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 >> v1:172.21.15.113:6789/0 conn(0x7f72840d1730 legacy=0x7f728409aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f72837fe700 1 --2- 172.21.15.113:0/1876766963 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7274003a00 0x7f7274003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f72837fe700 1 --2- 172.21.15.113:0/1876766963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7274004850 0x7f7274004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f72837fe700 1 --2- 172.21.15.113:0/1876766963 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7284161a70 0x7f7274005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72840d1c10 con 0x7f7274003a00 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72840b2ab0 con 0x7f7274004850 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f727400bb40 con 0x7f7284161a70 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f7288b96700 1 --2- 172.21.15.113:0/1876766963 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7274003a00 0x7f7274003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f7283fff700 1 --2- 172.21.15.113:0/1876766963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7274004850 0x7f7274004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f7289397700 1 --2- 172.21.15.113:0/1876766963 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7284161a70 0x7f7274005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f7288b96700 1 -- 172.21.15.113:0/1876766963 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7284161a70 msgr2=0x7f7274005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f7288b96700 1 --2- 172.21.15.113:0/1876766963 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7284161a70 0x7f7274005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f7288b96700 1 -- 172.21.15.113:0/1876766963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7274004850 msgr2=0x7f7274004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f7288b96700 1 --2- 172.21.15.113:0/1876766963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7274004850 0x7f7274004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f7288b96700 1 -- 172.21.15.113:0/1876766963 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72740035b0 con 0x7f7274003a00 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f7288b96700 1 --2- 172.21.15.113:0/1876766963 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7274003a00 0x7f7274003e20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7278009920 tx=0x7f7274005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bdba87ff4b999d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727800c070 con 0x7f7274003a00 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7278013610 con 0x7f7274003a00 2024-01-15T11:26:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.694+0000 7f72837fe700 1 -- 172.21.15.113:0/1876766963 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7278013980 con 0x7f7274003a00 2024-01-15T11:26:10.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.695+0000 7f728adfa700 1 -- 172.21.15.113:0/1876766963 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7274003a00 msgr2=0x7f7274003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:10.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.695+0000 7f728adfa700 1 --2- 172.21.15.113:0/1876766963 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7274003a00 0x7f7274003e20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7278009920 tx=0x7f7274005880 comp rx=0 tx=0).stop 2024-01-15T11:26:10.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.695+0000 7f7289397700 1 -- 172.21.15.113:0/1876766963 reap_dead start 2024-01-15T11:26:10.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.695+0000 7f728adfa700 1 -- 172.21.15.113:0/1876766963 shutdown_connections 2024-01-15T11:26:10.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.695+0000 7f728adfa700 1 -- 172.21.15.113:0/1876766963 >> 172.21.15.113:0/1876766963 conn(0x7f72840d1e50 msgr2=0x7f72840d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:10.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.695+0000 7f728adfa700 1 -- 172.21.15.113:0/1876766963 shutdown_connections 2024-01-15T11:26:10.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.695+0000 7f728adfa700 1 -- 172.21.15.113:0/1876766963 wait complete. 2024-01-15T11:26:10.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.696+0000 7f728adfa700 1 Processor -- start 2024-01-15T11:26:10.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.696+0000 7f728adfa700 1 -- start start 2024-01-15T11:26:10.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f728adfa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f728415e7d0 0x7f72840b6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f728adfa700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f728415ebb0 0x7f72840b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f728adfa700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f72840b4af0 0x7f72840b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f728adfa700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72840b6380 con 0x7f728415e7d0 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f728adfa700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72840b54c0 con 0x7f728415ebb0 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f728adfa700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72840cb310 con 0x7f72840b4af0 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f7288b96700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f728415e7d0 0x7f72840b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f7288b96700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f728415e7d0 0x7f72840b6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43590/0 (socket says 172.21.15.113:43590) 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f7288b96700 1 -- 172.21.15.113:0/2474820828 learned_addr learned my addr 172.21.15.113:0/2474820828 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f7283fff700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f728415ebb0 0x7f72840b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f7289397700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f72840b4af0 0x7f72840b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f7288b96700 1 -- 172.21.15.113:0/2474820828 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f72840b4af0 msgr2=0x7f72840b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f7288b96700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f72840b4af0 0x7f72840b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f7288b96700 1 -- 172.21.15.113:0/2474820828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f728415ebb0 msgr2=0x7f72840b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f7288b96700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f728415ebb0 0x7f72840b6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f7288b96700 1 -- 172.21.15.113:0/2474820828 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72840b5700 con 0x7f728415e7d0 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f7283fff700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f728415ebb0 0x7f72840b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.697+0000 7f7289397700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f72840b4af0 0x7f72840b4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.698+0000 7f7288b96700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f728415e7d0 0x7f72840b6500 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7278009f10 tx=0x7f72780047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.698+0000 7f72817fa700 1 -- 172.21.15.113:0/2474820828 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7278013b90 con 0x7f728415e7d0 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.698+0000 7f72817fa700 1 -- 172.21.15.113:0/2474820828 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7278013cf0 con 0x7f728415e7d0 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.698+0000 7f728adfa700 1 -- 172.21.15.113:0/2474820828 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72840b59e0 con 0x7f728415e7d0 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.698+0000 7f72817fa700 1 -- 172.21.15.113:0/2474820828 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72780038e0 con 0x7f728415e7d0 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.698+0000 7f728adfa700 1 -- 172.21.15.113:0/2474820828 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72840b5fa0 con 0x7f728415e7d0 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.699+0000 7f728adfa700 1 -- 172.21.15.113:0/2474820828 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728406c120 con 0x7f728415e7d0 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.699+0000 7f72817fa700 1 -- 172.21.15.113:0/2474820828 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7278003a40 con 0x7f728415e7d0 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.700+0000 7f72817fa700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f726806b350 0x7f726806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.700+0000 7f72817fa700 1 -- 172.21.15.113:0/2474820828 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f727808b050 con 0x7f728415e7d0 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.700+0000 7f7283fff700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f726806b350 0x7f726806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:10.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.700+0000 7f7283fff700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f726806b350 0x7f726806d810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f728415eff0 tx=0x7f7270008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:10.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.704+0000 7f72817fa700 1 -- 172.21.15.113:0/2474820828 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7278055fb0 con 0x7f728415e7d0 2024-01-15T11:26:10.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:10.936+0000 7f728adfa700 1 -- 172.21.15.113:0/2474820828 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7284160e70 con 0x7f728415e7d0 2024-01-15T11:26:11.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.536+0000 7f72817fa700 1 -- 172.21.15.113:0/2474820828 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7278013e60 con 0x7f728415e7d0 2024-01-15T11:26:11.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.540+0000 7f728adfa700 1 -- 172.21.15.113:0/2474820828 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f726806b350 msgr2=0x7f726806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:11.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.540+0000 7f728adfa700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f726806b350 0x7f726806d810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f728415eff0 tx=0x7f7270008040 comp rx=0 tx=0).stop 2024-01-15T11:26:11.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.540+0000 7f728adfa700 1 -- 172.21.15.113:0/2474820828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f728415e7d0 msgr2=0x7f72840b6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:11.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.540+0000 7f728adfa700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f728415e7d0 0x7f72840b6500 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7278009f10 tx=0x7f72780047c0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.540+0000 7f728adfa700 1 -- 172.21.15.113:0/2474820828 shutdown_connections 2024-01-15T11:26:11.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.540+0000 7f728adfa700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f726806b350 0x7f726806d810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.540+0000 7f728adfa700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f72840b4af0 0x7f72840b4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.540+0000 7f728adfa700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f728415e7d0 0x7f72840b6500 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.540+0000 7f728adfa700 1 --2- 172.21.15.113:0/2474820828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f728415ebb0 0x7f72840b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.540+0000 7f728adfa700 1 -- 172.21.15.113:0/2474820828 >> 172.21.15.113:0/2474820828 conn(0x7f72840d1e50 msgr2=0x7f728406efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:11.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.540+0000 7f728adfa700 1 -- 172.21.15.113:0/2474820828 shutdown_connections 2024-01-15T11:26:11.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.540+0000 7f728adfa700 1 -- 172.21.15.113:0/2474820828 wait complete. 2024-01-15T11:26:11.546 INFO:teuthology.orchestra.run.smithi113.stderr:new fs with metadata pool 4 and data pool 5 2024-01-15T11:26:11.569 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:26:11.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.691+0000 7fac5a6e0700 1 Processor -- start 2024-01-15T11:26:11.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.691+0000 7fac5a6e0700 1 -- start start 2024-01-15T11:26:11.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac5a6e0700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fac5415e080 0x7fac540b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:11.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac5a6e0700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fac540af4d0 0x7fac540afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:11.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac5a6e0700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac540ae430 con 0x7fac5406c000 2024-01-15T11:26:11.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac5a6e0700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac540af0b0 con 0x7fac540af4d0 2024-01-15T11:26:11.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac5a6e0700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac540b05a0 con 0x7fac5415e080 2024-01-15T11:26:11.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac537fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fac540af4d0 0x7fac540afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:11.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac537fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fac540af4d0 0x7fac540afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43602/0 (socket says 172.21.15.113:43602) 2024-01-15T11:26:11.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac537fe700 1 -- 172.21.15.113:0/157294829 learned_addr learned my addr 172.21.15.113:0/157294829 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:11.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac53fff700 1 --2- 172.21.15.113:0/157294829 >> v2:172.21.15.141:3300/0 conn(0x7fac5415e080 0x7fac540b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:11.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac537fe700 1 -- 172.21.15.113:0/157294829 >> v2:172.21.15.141:3300/0 conn(0x7fac5415e080 msgr2=0x7fac540b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:11.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac537fe700 1 --2- 172.21.15.113:0/157294829 >> v2:172.21.15.141:3300/0 conn(0x7fac5415e080 0x7fac540b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac537fe700 1 -- 172.21.15.113:0/157294829 >> v1:172.21.15.141:6789/0 conn(0x7fac5406c000 legacy=0x7fac54073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:11.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.692+0000 7fac537fe700 1 -- 172.21.15.113:0/157294829 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac540ae710 con 0x7fac540af4d0 2024-01-15T11:26:11.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.693+0000 7fac537fe700 1 --2- 172.21.15.113:0/157294829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac540af4d0 0x7fac540afb90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fac54072a90 tx=0x7fac48009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb0163f644aa7aa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:11.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.693+0000 7fac52ffd700 1 -- 172.21.15.113:0/157294829 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48004230 con 0x7fac540af4d0 2024-01-15T11:26:11.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.693+0000 7fac52ffd700 1 -- 172.21.15.113:0/157294829 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac48004390 con 0x7fac540af4d0 2024-01-15T11:26:11.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.693+0000 7fac52ffd700 1 -- 172.21.15.113:0/157294829 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac480036a0 con 0x7fac540af4d0 2024-01-15T11:26:11.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.693+0000 7fac5a6e0700 1 -- 172.21.15.113:0/157294829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac540af4d0 msgr2=0x7fac540afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:11.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.693+0000 7fac5a6e0700 1 --2- 172.21.15.113:0/157294829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac540af4d0 0x7fac540afb90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fac54072a90 tx=0x7fac48009c00 comp rx=0 tx=0).stop 2024-01-15T11:26:11.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.693+0000 7fac5a6e0700 1 -- 172.21.15.113:0/157294829 shutdown_connections 2024-01-15T11:26:11.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.693+0000 7fac5a6e0700 1 --2- 172.21.15.113:0/157294829 >> v2:172.21.15.141:3300/0 conn(0x7fac5415e080 0x7fac540b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.693+0000 7fac5a6e0700 1 --2- 172.21.15.113:0/157294829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac540af4d0 0x7fac540afb90 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.693+0000 7fac5a6e0700 1 -- 172.21.15.113:0/157294829 >> 172.21.15.113:0/157294829 conn(0x7fac54071b90 msgr2=0x7fac54071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:11.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.693+0000 7fac5a6e0700 1 -- 172.21.15.113:0/157294829 shutdown_connections 2024-01-15T11:26:11.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.693+0000 7fac5a6e0700 1 -- 172.21.15.113:0/157294829 wait complete. 2024-01-15T11:26:11.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.694+0000 7fac5a6e0700 1 Processor -- start 2024-01-15T11:26:11.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.695+0000 7fac5a6e0700 1 -- start start 2024-01-15T11:26:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.695+0000 7fac5a6e0700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac5406c000 0x7fac54151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.695+0000 7fac5a6e0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac540af4d0 0x7fac541f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.695+0000 7fac5a6e0700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac5415e080 0x7fac54157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.695+0000 7fac5a6e0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54151c60 con 0x7fac540af4d0 2024-01-15T11:26:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.695+0000 7fac5a6e0700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac541f4930 con 0x7fac5415e080 2024-01-15T11:26:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.695+0000 7fac5a6e0700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac54156b90 con 0x7fac5406c000 2024-01-15T11:26:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac537fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac540af4d0 0x7fac541f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac537fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac540af4d0 0x7fac541f43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43618/0 (socket says 172.21.15.113:43618) 2024-01-15T11:26:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac537fe700 1 -- 172.21.15.113:0/842634068 learned_addr learned my addr 172.21.15.113:0/842634068 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac53fff700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac5406c000 0x7fac54151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac58c7d700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac5415e080 0x7fac54157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac537fe700 1 -- 172.21.15.113:0/842634068 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac5406c000 msgr2=0x7fac54151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac537fe700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac5406c000 0x7fac54151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac537fe700 1 -- 172.21.15.113:0/842634068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac5415e080 msgr2=0x7fac54157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac537fe700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac5415e080 0x7fac54157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac537fe700 1 -- 172.21.15.113:0/842634068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac48009550 con 0x7fac540af4d0 2024-01-15T11:26:11.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac53fff700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac5406c000 0x7fac54151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:11.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac58c7d700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac5415e080 0x7fac54157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:11.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac537fe700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac540af4d0 0x7fac541f43f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fac48009870 tx=0x7fac48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:11.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac50ff9700 1 -- 172.21.15.113:0/842634068 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48003d80 con 0x7fac540af4d0 2024-01-15T11:26:11.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac50ff9700 1 -- 172.21.15.113:0/842634068 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4801f070 con 0x7fac540af4d0 2024-01-15T11:26:11.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac5a6e0700 1 -- 172.21.15.113:0/842634068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac54155f80 con 0x7fac540af4d0 2024-01-15T11:26:11.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac50ff9700 1 -- 172.21.15.113:0/842634068 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48011850 con 0x7fac540af4d0 2024-01-15T11:26:11.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.696+0000 7fac5a6e0700 1 -- 172.21.15.113:0/842634068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac54156540 con 0x7fac540af4d0 2024-01-15T11:26:11.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.697+0000 7fac5a6e0700 1 -- 172.21.15.113:0/842634068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac540753f0 con 0x7fac540af4d0 2024-01-15T11:26:11.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.697+0000 7fac50ff9700 1 -- 172.21.15.113:0/842634068 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fac48021030 con 0x7fac540af4d0 2024-01-15T11:26:11.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.698+0000 7fac50ff9700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fac4006b350 0x7fac4006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:11.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.698+0000 7fac50ff9700 1 -- 172.21.15.113:0/842634068 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac4808b110 con 0x7fac540af4d0 2024-01-15T11:26:11.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.698+0000 7fac53fff700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fac4006b350 0x7fac4006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:11.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.698+0000 7fac53fff700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fac4006b350 0x7fac4006d810 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fac44007a50 tx=0x7fac44008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:11.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.702+0000 7fac50ff9700 1 -- 172.21.15.113:0/842634068 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac48056010 con 0x7fac540af4d0 2024-01-15T11:26:11.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.894+0000 7fac5a6e0700 1 -- 172.21.15.113:0/842634068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac540753f0 con 0x7fac540af4d0 2024-01-15T11:26:11.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.895+0000 7fac50ff9700 1 -- 172.21.15.113:0/842634068 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16642 (secure 0 0 0) 0x7fac48059630 con 0x7fac540af4d0 2024-01-15T11:26:11.895 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:11.895 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":32,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:26:10.938173+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:26:08.840011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-15T11:26:09.908268+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:26:11.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.897+0000 7fac5a6e0700 1 -- 172.21.15.113:0/842634068 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fac4006b350 msgr2=0x7fac4006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:11.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.897+0000 7fac5a6e0700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fac4006b350 0x7fac4006d810 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fac44007a50 tx=0x7fac44008040 comp rx=0 tx=0).stop 2024-01-15T11:26:11.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.897+0000 7fac5a6e0700 1 -- 172.21.15.113:0/842634068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac540af4d0 msgr2=0x7fac541f43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:11.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.897+0000 7fac5a6e0700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac540af4d0 0x7fac541f43f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fac48009870 tx=0x7fac48004750 comp rx=0 tx=0).stop 2024-01-15T11:26:11.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.897+0000 7fac5a6e0700 1 -- 172.21.15.113:0/842634068 shutdown_connections 2024-01-15T11:26:11.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.897+0000 7fac5a6e0700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fac4006b350 0x7fac4006d810 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.897+0000 7fac5a6e0700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fac5415e080 0x7fac54157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.897+0000 7fac5a6e0700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fac5406c000 0x7fac54151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.897+0000 7fac5a6e0700 1 --2- 172.21.15.113:0/842634068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac540af4d0 0x7fac541f43f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:11.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.897+0000 7fac5a6e0700 1 -- 172.21.15.113:0/842634068 >> 172.21.15.113:0/842634068 conn(0x7fac54071b90 msgr2=0x7fac5415d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:11.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.898+0000 7fac5a6e0700 1 -- 172.21.15.113:0/842634068 shutdown_connections 2024-01-15T11:26:11.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:11.898+0000 7fac5a6e0700 1 -- 172.21.15.113:0/842634068 wait complete. 2024-01-15T11:26:11.909 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:26:12.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.032+0000 7fd9e96be700 1 Processor -- start 2024-01-15T11:26:12.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.032+0000 7fd9e96be700 1 -- start start 2024-01-15T11:26:12.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.032+0000 7fd9e96be700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd9e415e1e0 0x7fd9e40d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:12.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.032+0000 7fd9e96be700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd9e40d3a00 0x7fd9e40d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:12.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.032+0000 7fd9e96be700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9e40d0db0 0x7fd9e406aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:12.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.032+0000 7fd9e96be700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e40d3150 con 0x7fd9e415e1e0 2024-01-15T11:26:12.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.032+0000 7fd9e96be700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e40d1240 con 0x7fd9e40d3a00 2024-01-15T11:26:12.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.032+0000 7fd9e96be700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e415e5c0 con 0x7fd9e40d0db0 2024-01-15T11:26:12.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e2ffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd9e415e1e0 0x7fd9e40d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:12.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e2ffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd9e415e1e0 0x7fd9e40d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43622/0 (socket says 172.21.15.113:43622) 2024-01-15T11:26:12.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e2ffd700 1 -- 172.21.15.113:0/2382154345 learned_addr learned my addr 172.21.15.113:0/2382154345 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:12.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e37fe700 1 --2- 172.21.15.113:0/2382154345 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9e40d0db0 0x7fd9e406aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:12.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e27fc700 1 --2- 172.21.15.113:0/2382154345 >> v2:172.21.15.141:3300/0 conn(0x7fd9e40d3a00 0x7fd9e40d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:12.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e2ffd700 1 -- 172.21.15.113:0/2382154345 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9e40d0db0 msgr2=0x7fd9e406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:12.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e2ffd700 1 --2- 172.21.15.113:0/2382154345 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9e40d0db0 0x7fd9e406aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e2ffd700 1 -- 172.21.15.113:0/2382154345 >> v2:172.21.15.141:3300/0 conn(0x7fd9e40d3a00 msgr2=0x7fd9e40d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:12.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e2ffd700 1 --2- 172.21.15.113:0/2382154345 >> v2:172.21.15.141:3300/0 conn(0x7fd9e40d3a00 0x7fd9e40d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e2ffd700 1 -- 172.21.15.113:0/2382154345 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e40d20b0 con 0x7fd9e415e1e0 2024-01-15T11:26:12.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e27fc700 1 --2- 172.21.15.113:0/2382154345 >> v2:172.21.15.141:3300/0 conn(0x7fd9e40d3a00 0x7fd9e40d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:12.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e2ffd700 1 --2- 172.21.15.113:0/2382154345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9e415e1e0 0x7fd9e40d34c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4161ba0 tx=0x7fd9d4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08764ac6faa73b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:12.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e1ffb700 1 -- 172.21.15.113:0/2382154345 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d400b860 con 0x7fd9e415e1e0 2024-01-15T11:26:12.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e1ffb700 1 -- 172.21.15.113:0/2382154345 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d400b9c0 con 0x7fd9e415e1e0 2024-01-15T11:26:12.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.033+0000 7fd9e1ffb700 1 -- 172.21.15.113:0/2382154345 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d400bd50 con 0x7fd9e415e1e0 2024-01-15T11:26:12.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.034+0000 7fd9e96be700 1 -- 172.21.15.113:0/2382154345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9e415e1e0 msgr2=0x7fd9e40d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:12.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.034+0000 7fd9e96be700 1 --2- 172.21.15.113:0/2382154345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9e415e1e0 0x7fd9e40d34c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4161ba0 tx=0x7fd9d4009c00 comp rx=0 tx=0).stop 2024-01-15T11:26:12.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.034+0000 7fd9e96be700 1 -- 172.21.15.113:0/2382154345 shutdown_connections 2024-01-15T11:26:12.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.034+0000 7fd9e96be700 1 --2- 172.21.15.113:0/2382154345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9e415e1e0 0x7fd9e40d34c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.034+0000 7fd9e96be700 1 --2- 172.21.15.113:0/2382154345 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9e40d0db0 0x7fd9e406aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.034+0000 7fd9e96be700 1 --2- 172.21.15.113:0/2382154345 >> v2:172.21.15.141:3300/0 conn(0x7fd9e40d3a00 0x7fd9e40d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.034+0000 7fd9e96be700 1 -- 172.21.15.113:0/2382154345 >> 172.21.15.113:0/2382154345 conn(0x7fd9e40b5b50 msgr2=0x7fd9e40b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:12.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.034+0000 7fd9e96be700 1 -- 172.21.15.113:0/2382154345 shutdown_connections 2024-01-15T11:26:12.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.034+0000 7fd9e96be700 1 -- 172.21.15.113:0/2382154345 wait complete. 2024-01-15T11:26:12.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.035+0000 7fd9e96be700 1 Processor -- start 2024-01-15T11:26:12.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.036+0000 7fd9e96be700 1 -- start start 2024-01-15T11:26:12.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.036+0000 7fd9e96be700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9e40d0db0 0x7fd9e40cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:12.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.036+0000 7fd9e96be700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9e40d3a00 0x7fd9e414a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:12.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.036+0000 7fd9e96be700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9e415e1e0 0x7fd9e41f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:12.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.036+0000 7fd9e96be700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e414a940 con 0x7fd9e415e1e0 2024-01-15T11:26:12.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.036+0000 7fd9e96be700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e4157690 con 0x7fd9e40d3a00 2024-01-15T11:26:12.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.036+0000 7fd9e96be700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e41578e0 con 0x7fd9e40d0db0 2024-01-15T11:26:12.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9e415e1e0 0x7fd9e41f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:12.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9e415e1e0 0x7fd9e41f3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43630/0 (socket says 172.21.15.113:43630) 2024-01-15T11:26:12.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e37fe700 1 -- 172.21.15.113:0/225931127 learned_addr learned my addr 172.21.15.113:0/225931127 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:12.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e27fc700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9e40d3a00 0x7fd9e414a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:12.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e2ffd700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9e40d0db0 0x7fd9e40cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:12.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e37fe700 1 -- 172.21.15.113:0/225931127 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9e40d0db0 msgr2=0x7fd9e40cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:12.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e37fe700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9e40d0db0 0x7fd9e40cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e37fe700 1 -- 172.21.15.113:0/225931127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9e40d3a00 msgr2=0x7fd9e414a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:12.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e37fe700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9e40d3a00 0x7fd9e414a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e37fe700 1 -- 172.21.15.113:0/225931127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d4009550 con 0x7fd9e415e1e0 2024-01-15T11:26:12.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e37fe700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9e415e1e0 0x7fd9e41f3200 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc007e40 tx=0x7fd9dc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:12.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9db7fe700 1 -- 172.21.15.113:0/225931127 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc00eea0 con 0x7fd9e415e1e0 2024-01-15T11:26:12.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9db7fe700 1 -- 172.21.15.113:0/225931127 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc014420 con 0x7fd9e415e1e0 2024-01-15T11:26:12.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e96be700 1 -- 172.21.15.113:0/225931127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e4157bc0 con 0x7fd9e415e1e0 2024-01-15T11:26:12.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9db7fe700 1 -- 172.21.15.113:0/225931127 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc0147b0 con 0x7fd9e415e1e0 2024-01-15T11:26:12.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.037+0000 7fd9e96be700 1 -- 172.21.15.113:0/225931127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e40cb2b0 con 0x7fd9e415e1e0 2024-01-15T11:26:12.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.038+0000 7fd9e96be700 1 -- 172.21.15.113:0/225931127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e406bea0 con 0x7fd9e415e1e0 2024-01-15T11:26:12.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.038+0000 7fd9db7fe700 1 -- 172.21.15.113:0/225931127 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd9dc014910 con 0x7fd9e415e1e0 2024-01-15T11:26:12.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.039+0000 7fd9db7fe700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd9c406b350 0x7fd9c406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:12.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.039+0000 7fd9db7fe700 1 -- 172.21.15.113:0/225931127 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9dc08a3e0 con 0x7fd9e415e1e0 2024-01-15T11:26:12.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.039+0000 7fd9e2ffd700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd9c406b350 0x7fd9c406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:12.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.039+0000 7fd9e2ffd700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd9c406b350 0x7fd9c406d810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4000c00 tx=0x7fd9d4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:12.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.043+0000 7fd9db7fe700 1 -- 172.21.15.113:0/225931127 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9dc055250 con 0x7fd9e415e1e0 2024-01-15T11:26:12.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.237+0000 7fd9e96be700 1 -- 172.21.15.113:0/225931127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9e40751e0 con 0x7fd9e415e1e0 2024-01-15T11:26:12.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.238+0000 7fd9db7fe700 1 -- 172.21.15.113:0/225931127 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16642 (secure 0 0 0) 0x7fd9dc058870 con 0x7fd9e415e1e0 2024-01-15T11:26:12.238 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:12.238 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":32,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:26:10.938173+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:26:08.840011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-15T11:26:09.908268+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:26:12.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.240+0000 7fd9e96be700 1 -- 172.21.15.113:0/225931127 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd9c406b350 msgr2=0x7fd9c406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:12.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.240+0000 7fd9e96be700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd9c406b350 0x7fd9c406d810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4000c00 tx=0x7fd9d4008030 comp rx=0 tx=0).stop 2024-01-15T11:26:12.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.240+0000 7fd9e96be700 1 -- 172.21.15.113:0/225931127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9e415e1e0 msgr2=0x7fd9e41f3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:12.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.240+0000 7fd9e96be700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9e415e1e0 0x7fd9e41f3200 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc007e40 tx=0x7fd9dc00e3c0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.240+0000 7fd9e96be700 1 -- 172.21.15.113:0/225931127 shutdown_connections 2024-01-15T11:26:12.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.240+0000 7fd9e96be700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd9c406b350 0x7fd9c406d810 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.240+0000 7fd9e96be700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9e415e1e0 0x7fd9e41f3200 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.240+0000 7fd9e96be700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9e40d0db0 0x7fd9e40cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.240+0000 7fd9e96be700 1 --2- 172.21.15.113:0/225931127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9e40d3a00 0x7fd9e414a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.240+0000 7fd9e96be700 1 -- 172.21.15.113:0/225931127 >> 172.21.15.113:0/225931127 conn(0x7fd9e40b5b50 msgr2=0x7fd9e415d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:12.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.240+0000 7fd9e96be700 1 -- 172.21.15.113:0/225931127 shutdown_connections 2024-01-15T11:26:12.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.240+0000 7fd9e96be700 1 -- 172.21.15.113:0/225931127 wait complete. 2024-01-15T11:26:12.252 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T11:26:12.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.378+0000 7fce5dcde700 1 Processor -- start 2024-01-15T11:26:12.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.378+0000 7fce5dcde700 1 -- start start 2024-01-15T11:26:12.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.379+0000 7fce5dcde700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fce580d03a0 0x7fce580b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:12.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.379+0000 7fce5dcde700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce580b3860 con 0x7fce580cffc0 2024-01-15T11:26:12.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.379+0000 7fce5dcde700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce580b2520 con 0x7fce5815f8d0 2024-01-15T11:26:12.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.379+0000 7fce5dcde700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce580b2800 con 0x7fce580d03a0 2024-01-15T11:26:12.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.379+0000 7fce57fff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fce580d03a0 0x7fce580b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:12.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.379+0000 7fce577fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fce580cffc0 0x7fce5815f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:43488/0 (socket says 172.21.15.113:43488) 2024-01-15T11:26:12.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.379+0000 7fce577fe700 1 -- 172.21.15.113:0/3515264580 learned_addr learned my addr 172.21.15.113:0/3515264580 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:12.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.379+0000 7fce57fff700 1 -- 172.21.15.113:0/3515264580 >> v1:172.21.15.141:6789/0 conn(0x7fce5815f8d0 legacy=0x7fce580ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:12.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.379+0000 7fce57fff700 1 -- 172.21.15.113:0/3515264580 >> v1:172.21.15.113:6789/0 conn(0x7fce580cffc0 legacy=0x7fce5815f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:12.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.379+0000 7fce57fff700 1 -- 172.21.15.113:0/3515264580 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce580b2ae0 con 0x7fce580d03a0 2024-01-15T11:26:12.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.379+0000 7fce57fff700 1 --2- 172.21.15.113:0/3515264580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce580d03a0 0x7fce580b3320 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fce50009d90 tx=0x7fce5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afe698cfd8fd2b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:12.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.379+0000 7fce567fc700 1 -- 172.21.15.113:0/3515264580 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5000e070 con 0x7fce580d03a0 2024-01-15T11:26:12.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.380+0000 7fce567fc700 1 -- 172.21.15.113:0/3515264580 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5000bb90 con 0x7fce580d03a0 2024-01-15T11:26:12.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.380+0000 7fce567fc700 1 -- 172.21.15.113:0/3515264580 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50003a20 con 0x7fce580d03a0 2024-01-15T11:26:12.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.380+0000 7fce5dcde700 1 -- 172.21.15.113:0/3515264580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce580d03a0 msgr2=0x7fce580b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:12.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.380+0000 7fce5dcde700 1 --2- 172.21.15.113:0/3515264580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce580d03a0 0x7fce580b3320 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fce50009d90 tx=0x7fce5000b330 comp rx=0 tx=0).stop 2024-01-15T11:26:12.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.380+0000 7fce5dcde700 1 -- 172.21.15.113:0/3515264580 shutdown_connections 2024-01-15T11:26:12.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.380+0000 7fce5dcde700 1 --2- 172.21.15.113:0/3515264580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce580d03a0 0x7fce580b3320 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.380+0000 7fce5dcde700 1 -- 172.21.15.113:0/3515264580 >> 172.21.15.113:0/3515264580 conn(0x7fce580d1e50 msgr2=0x7fce580d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:12.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.380+0000 7fce5dcde700 1 -- 172.21.15.113:0/3515264580 shutdown_connections 2024-01-15T11:26:12.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.380+0000 7fce5dcde700 1 -- 172.21.15.113:0/3515264580 wait complete. 2024-01-15T11:26:12.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.381+0000 7fce5dcde700 1 Processor -- start 2024-01-15T11:26:12.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.381+0000 7fce5dcde700 1 -- start start 2024-01-15T11:26:12.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.382+0000 7fce5dcde700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fce580cffc0 0x7fce58156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:12.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.382+0000 7fce5dcde700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce580d03a0 0x7fce580b5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:12.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.382+0000 7fce5dcde700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce5815f8d0 0x7fce580b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:12.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.382+0000 7fce5dcde700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce58156f90 con 0x7fce5815f8d0 2024-01-15T11:26:12.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.382+0000 7fce5dcde700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce580b55b0 con 0x7fce580d03a0 2024-01-15T11:26:12.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.382+0000 7fce5dcde700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce580b5890 con 0x7fce580cffc0 2024-01-15T11:26:12.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.382+0000 7fce57fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce5815f8d0 0x7fce580b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:12.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.382+0000 7fce57fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce5815f8d0 0x7fce580b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43638/0 (socket says 172.21.15.113:43638) 2024-01-15T11:26:12.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.382+0000 7fce57fff700 1 -- 172.21.15.113:0/33575770 learned_addr learned my addr 172.21.15.113:0/33575770 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:12.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.383+0000 7fce57fff700 1 -- 172.21.15.113:0/33575770 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fce580cffc0 msgr2=0x7fce58156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:12.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.383+0000 7fce57fff700 1 --2- 172.21.15.113:0/33575770 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fce580cffc0 0x7fce58156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.383+0000 7fce57fff700 1 -- 172.21.15.113:0/33575770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce580d03a0 msgr2=0x7fce580b5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:12.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.383+0000 7fce57fff700 1 --2- 172.21.15.113:0/33575770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce580d03a0 0x7fce580b5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:12.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.383+0000 7fce57fff700 1 -- 172.21.15.113:0/33575770 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce50009a70 con 0x7fce5815f8d0 2024-01-15T11:26:12.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.383+0000 7fce57fff700 1 --2- 172.21.15.113:0/33575770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce5815f8d0 0x7fce580b4af0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fce58072870 tx=0x7fce50004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:12.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.383+0000 7fce4ffff700 1 -- 172.21.15.113:0/33575770 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50012630 con 0x7fce5815f8d0 2024-01-15T11:26:12.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.383+0000 7fce4ffff700 1 -- 172.21.15.113:0/33575770 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce500127b0 con 0x7fce5815f8d0 2024-01-15T11:26:12.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.383+0000 7fce5dcde700 1 -- 172.21.15.113:0/33575770 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5815a8e0 con 0x7fce5815f8d0 2024-01-15T11:26:12.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.383+0000 7fce4ffff700 1 -- 172.21.15.113:0/33575770 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50012b60 con 0x7fce5815f8d0 2024-01-15T11:26:12.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.383+0000 7fce5dcde700 1 -- 172.21.15.113:0/33575770 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce580c6310 con 0x7fce5815f8d0 2024-01-15T11:26:12.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.384+0000 7fce5dcde700 1 -- 172.21.15.113:0/33575770 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5815a740 con 0x7fce5815f8d0 2024-01-15T11:26:12.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.384+0000 7fce4ffff700 1 -- 172.21.15.113:0/33575770 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fce50012cc0 con 0x7fce5815f8d0 2024-01-15T11:26:12.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.385+0000 7fce4ffff700 1 --2- 172.21.15.113:0/33575770 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fce3806b350 0x7fce3806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:12.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.385+0000 7fce4ffff700 1 -- 172.21.15.113:0/33575770 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce5008c310 con 0x7fce5815f8d0 2024-01-15T11:26:12.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.385+0000 7fce577fe700 1 --2- 172.21.15.113:0/33575770 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fce3806b350 0x7fce3806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:12.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.385+0000 7fce577fe700 1 --2- 172.21.15.113:0/33575770 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fce3806b350 0x7fce3806d810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fce40000f60 tx=0x7fce40007b10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:12.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.389+0000 7fce4ffff700 1 -- 172.21.15.113:0/33575770 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce5815a740 con 0x7fce5815f8d0 2024-01-15T11:26:12.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:12.630+0000 7fce5dcde700 1 -- 172.21.15.113:0/33575770 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fce5815ae00 con 0x7fce5815f8d0 2024-01-15T11:26:13.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.570+0000 7fce4ffff700 1 -- 172.21.15.113:0/33575770 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7fce5815ae00 con 0x7fce5815f8d0 2024-01-15T11:26:13.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.573+0000 7fce5dcde700 1 -- 172.21.15.113:0/33575770 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fce3806b350 msgr2=0x7fce3806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:13.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.573+0000 7fce5dcde700 1 --2- 172.21.15.113:0/33575770 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fce3806b350 0x7fce3806d810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fce40000f60 tx=0x7fce40007b10 comp rx=0 tx=0).stop 2024-01-15T11:26:13.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.573+0000 7fce5dcde700 1 -- 172.21.15.113:0/33575770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce5815f8d0 msgr2=0x7fce580b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:13.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.573+0000 7fce5dcde700 1 --2- 172.21.15.113:0/33575770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce5815f8d0 0x7fce580b4af0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fce58072870 tx=0x7fce50004ad0 comp rx=0 tx=0).stop 2024-01-15T11:26:13.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.573+0000 7fce5dcde700 1 -- 172.21.15.113:0/33575770 shutdown_connections 2024-01-15T11:26:13.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.573+0000 7fce5dcde700 1 --2- 172.21.15.113:0/33575770 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fce3806b350 0x7fce3806d810 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:13.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.573+0000 7fce5dcde700 1 --2- 172.21.15.113:0/33575770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce5815f8d0 0x7fce580b4af0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:13.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.573+0000 7fce5dcde700 1 --2- 172.21.15.113:0/33575770 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fce580cffc0 0x7fce58156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:13.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.573+0000 7fce5dcde700 1 --2- 172.21.15.113:0/33575770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fce580d03a0 0x7fce580b5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:13.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.573+0000 7fce5dcde700 1 -- 172.21.15.113:0/33575770 >> 172.21.15.113:0/33575770 conn(0x7fce580d1e50 msgr2=0x7fce5815d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:13.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.573+0000 7fce5dcde700 1 -- 172.21.15.113:0/33575770 shutdown_connections 2024-01-15T11:26:13.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.573+0000 7fce5dcde700 1 -- 172.21.15.113:0/33575770 wait complete. 2024-01-15T11:26:13.587 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T11:26:13.587 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T11:26:13.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.720+0000 7f1592ada700 1 Processor -- start 2024-01-15T11:26:13.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.720+0000 7f1592ada700 1 -- start start 2024-01-15T11:26:13.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f1592ada700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f158c0cffc0 0x7f158c0b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:13.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f1592ada700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158c0b3860 con 0x7f158c0d03a0 2024-01-15T11:26:13.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f1592ada700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158c0b2520 con 0x7f158c15f8d0 2024-01-15T11:26:13.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f1592ada700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f158c0b2800 con 0x7f158c0cffc0 2024-01-15T11:26:13.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158bfff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f158c0d03a0 0x7f158c15f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:43496/0 (socket says 172.21.15.113:43496) 2024-01-15T11:26:13.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158bfff700 1 -- 172.21.15.113:0/1709457013 learned_addr learned my addr 172.21.15.113:0/1709457013 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:13.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744959043 0 0) 0x7f1578001110 con 0x7f158c0d03a0 2024-01-15T11:26:13.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f1590876700 1 --2- 172.21.15.113:0/1709457013 >> v2:172.21.15.141:3300/0 conn(0x7f158c0cffc0 0x7f158c0b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:13.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f157c0035b0 con 0x7f158c0d03a0 2024-01-15T11:26:13.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4036464436 0 0) 0x7f157c0035b0 con 0x7f158c0d03a0 2024-01-15T11:26:13.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 >> v2:172.21.15.141:3300/0 conn(0x7f158c0cffc0 msgr2=0x7f158c0b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 --2- 172.21.15.113:0/1709457013 >> v2:172.21.15.141:3300/0 conn(0x7f158c0cffc0 0x7f158c0b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 >> v1:172.21.15.141:6789/0 conn(0x7f158c15f8d0 legacy=0x7f158c0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158c0b2ae0 con 0x7f158c0d03a0 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f1578001800 con 0x7f158c0d03a0 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 >> v1:172.21.15.113:6789/0 conn(0x7f158c0d03a0 legacy=0x7f158c15f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 --2- 172.21.15.113:0/1709457013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 0x7f157c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 --2- 172.21.15.113:0/1709457013 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f157c004850 0x7f157c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 --2- 172.21.15.113:0/1709457013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f157c005210 0x7f157c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158c0b2800 con 0x7f157c003a00 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157c00c1b0 con 0x7f157c005210 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.721+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157c00c370 con 0x7f157c004850 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f1590876700 1 --2- 172.21.15.113:0/1709457013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 0x7f157c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f1591077700 1 --2- 172.21.15.113:0/1709457013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f157c005210 0x7f157c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:13.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f158bfff700 1 --2- 172.21.15.113:0/1709457013 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f157c004850 0x7f157c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f1590876700 1 -- 172.21.15.113:0/1709457013 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f157c004850 msgr2=0x7f157c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f1590876700 1 --2- 172.21.15.113:0/1709457013 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f157c004850 0x7f157c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f1590876700 1 -- 172.21.15.113:0/1709457013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f157c005210 msgr2=0x7f157c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f1590876700 1 --2- 172.21.15.113:0/1709457013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f157c005210 0x7f157c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f1590876700 1 -- 172.21.15.113:0/1709457013 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c0035b0 con 0x7f157c003a00 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f1591077700 1 --2- 172.21.15.113:0/1709457013 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f157c005210 0x7f157c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f1591077700 1 -- 172.21.15.113:0/1709457013 reap_dead start 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f1590876700 1 --2- 172.21.15.113:0/1709457013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 0x7f157c003e20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1580009b60 tx=0x7f157c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9dd93597c8dfa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158000e6c0 con 0x7f157c003a00 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158000e820 con 0x7f157c003a00 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.722+0000 7f158b7fe700 1 -- 172.21.15.113:0/1709457013 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158000eb90 con 0x7f157c003a00 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.723+0000 7f1592ada700 1 -- 172.21.15.113:0/1709457013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 msgr2=0x7f157c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.723+0000 7f1592ada700 1 --2- 172.21.15.113:0/1709457013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 0x7f157c003e20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1580009b60 tx=0x7f157c005880 comp rx=0 tx=0).stop 2024-01-15T11:26:13.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.723+0000 7f1592ada700 1 -- 172.21.15.113:0/1709457013 shutdown_connections 2024-01-15T11:26:13.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.723+0000 7f1592ada700 1 --2- 172.21.15.113:0/1709457013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 0x7f157c003e20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:13.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.723+0000 7f1592ada700 1 -- 172.21.15.113:0/1709457013 >> 172.21.15.113:0/1709457013 conn(0x7f158c0d1e50 msgr2=0x7f158c0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:13.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.723+0000 7f1592ada700 1 -- 172.21.15.113:0/1709457013 shutdown_connections 2024-01-15T11:26:13.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.723+0000 7f1592ada700 1 -- 172.21.15.113:0/1709457013 wait complete. 2024-01-15T11:26:13.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.724+0000 7f1592ada700 1 Processor -- start 2024-01-15T11:26:13.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.724+0000 7f1592ada700 1 -- start start 2024-01-15T11:26:13.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1592ada700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 0x7f158c155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1592ada700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f158c14c9b0 0x7f158c14ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1592ada700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f158c14d370 0x7f158c156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1592ada700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158c0b4af0 con 0x7f157c003a00 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1592ada700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158c0b4d40 con 0x7f158c14c9b0 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1592ada700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f158c0b5020 con 0x7f158c14d370 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1590876700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 0x7f158c155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1590876700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 0x7f158c155280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43660/0 (socket says 172.21.15.113:43660) 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1590876700 1 -- 172.21.15.113:0/1205807748 learned_addr learned my addr 172.21.15.113:0/1205807748 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1591077700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f158c14d370 0x7f158c156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f158bfff700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f158c14c9b0 0x7f158c14ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1590876700 1 -- 172.21.15.113:0/1205807748 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f158c14d370 msgr2=0x7f158c156ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1590876700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f158c14d370 0x7f158c156ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1590876700 1 -- 172.21.15.113:0/1205807748 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f158c14c9b0 msgr2=0x7f158c14ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:13.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1590876700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f158c14c9b0 0x7f158c14ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1590876700 1 -- 172.21.15.113:0/1205807748 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158c0b5300 con 0x7f157c003a00 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1590876700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 0x7f158c155280 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f157c00c180 tx=0x7f15800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f15897fa700 1 -- 172.21.15.113:0/1205807748 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158000e6c0 con 0x7f157c003a00 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f15897fa700 1 -- 172.21.15.113:0/1205807748 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1580003ad0 con 0x7f157c003a00 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1592ada700 1 -- 172.21.15.113:0/1205807748 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158c0b55e0 con 0x7f157c003a00 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f15897fa700 1 -- 172.21.15.113:0/1205807748 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1580003e80 con 0x7f157c003a00 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.725+0000 7f1592ada700 1 -- 172.21.15.113:0/1205807748 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158c1559a0 con 0x7f157c003a00 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.726+0000 7f1592ada700 1 -- 172.21.15.113:0/1205807748 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f158c14d750 con 0x7f157c003a00 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.727+0000 7f15897fa700 1 -- 172.21.15.113:0/1205807748 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1580013470 con 0x7f157c003a00 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.727+0000 7f15897fa700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f157006b380 0x7f157006d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.727+0000 7f15897fa700 1 -- 172.21.15.113:0/1205807748 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f158008b520 con 0x7f157c003a00 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.727+0000 7f158bfff700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f157006b380 0x7f157006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.728+0000 7f158bfff700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f157006b380 0x7f157006d840 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1578001270 tx=0x7f1578009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.732+0000 7f15897fa700 1 -- 172.21.15.113:0/1205807748 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1580056540 con 0x7f157c003a00 2024-01-15T11:26:13.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:13.969+0000 7f1592ada700 1 -- 172.21.15.113:0/1205807748 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f158c1559a0 con 0x7f157c003a00 2024-01-15T11:26:14.199 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:26:14.202 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:26:14.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.576+0000 7f15897fa700 1 -- 172.21.15.113:0/1205807748 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f158c1559a0 con 0x7f157c003a00 2024-01-15T11:26:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.578+0000 7f1592ada700 1 -- 172.21.15.113:0/1205807748 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f157006b380 msgr2=0x7f157006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.578+0000 7f1592ada700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f157006b380 0x7f157006d840 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1578001270 tx=0x7f1578009040 comp rx=0 tx=0).stop 2024-01-15T11:26:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.578+0000 7f1592ada700 1 -- 172.21.15.113:0/1205807748 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 msgr2=0x7f158c155280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.578+0000 7f1592ada700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 0x7f158c155280 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f157c00c180 tx=0x7f15800047c0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.578+0000 7f1592ada700 1 -- 172.21.15.113:0/1205807748 shutdown_connections 2024-01-15T11:26:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.578+0000 7f1592ada700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f157006b380 0x7f157006d840 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.578+0000 7f1592ada700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f158c14d370 0x7f158c156ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.578+0000 7f1592ada700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f157c003a00 0x7f158c155280 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.578+0000 7f1592ada700 1 --2- 172.21.15.113:0/1205807748 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f158c14c9b0 0x7f158c14ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.578+0000 7f1592ada700 1 -- 172.21.15.113:0/1205807748 >> 172.21.15.113:0/1205807748 conn(0x7f158c0d1e50 msgr2=0x7f158c0ccc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.578+0000 7f1592ada700 1 -- 172.21.15.113:0/1205807748 shutdown_connections 2024-01-15T11:26:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.578+0000 7f1592ada700 1 -- 172.21.15.113:0/1205807748 wait complete. 2024-01-15T11:26:14.590 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T11:26:14.590 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T11:26:14.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.713+0000 7f105ba62700 1 Processor -- start 2024-01-15T11:26:14.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.713+0000 7f105ba62700 1 -- start start 2024-01-15T11:26:14.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f105ba62700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f105415bed0 0x7f10540b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:14.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f105ba62700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10541603c0 0x7f10540d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:14.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f105ba62700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f10540d1a50 0x7f10540d1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:14.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f105ba62700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f105415c2b0 con 0x7f10540d1a50 2024-01-15T11:26:14.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f105ba62700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10540d0770 con 0x7f105415bed0 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f105ba62700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10540cd100 con 0x7f10541603c0 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f1059fff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f10540d1a50 0x7f10540d1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f1059fff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f10540d1a50 0x7f10540d1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51950/0 (socket says 172.21.15.113:51950) 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f1059fff700 1 -- 172.21.15.113:0/3298574714 learned_addr learned my addr 172.21.15.113:0/3298574714 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f1058ffd700 1 --2- 172.21.15.113:0/3298574714 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10541603c0 0x7f10540d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f10597fe700 1 --2- 172.21.15.113:0/3298574714 >> v2:172.21.15.141:3300/0 conn(0x7f105415bed0 0x7f10540b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f1059fff700 1 -- 172.21.15.113:0/3298574714 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10541603c0 msgr2=0x7f10540d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f1059fff700 1 --2- 172.21.15.113:0/3298574714 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10541603c0 0x7f10540d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f1059fff700 1 -- 172.21.15.113:0/3298574714 >> v2:172.21.15.141:3300/0 conn(0x7f105415bed0 msgr2=0x7f10540b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f1059fff700 1 --2- 172.21.15.113:0/3298574714 >> v2:172.21.15.141:3300/0 conn(0x7f105415bed0 0x7f10540b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f1059fff700 1 -- 172.21.15.113:0/3298574714 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10540d2420 con 0x7f10540d1a50 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f1059fff700 1 --2- 172.21.15.113:0/3298574714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10540d1a50 0x7f10540d1ee0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f105409b2a0 tx=0x7f104c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c96e97a1d1aa2e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f104bfff700 1 -- 172.21.15.113:0/3298574714 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c00e070 con 0x7f10540d1a50 2024-01-15T11:26:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f104bfff700 1 -- 172.21.15.113:0/3298574714 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c00b9c0 con 0x7f10540d1a50 2024-01-15T11:26:14.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.714+0000 7f104bfff700 1 -- 172.21.15.113:0/3298574714 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c00bd70 con 0x7f10540d1a50 2024-01-15T11:26:14.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.715+0000 7f105ba62700 1 -- 172.21.15.113:0/3298574714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10540d1a50 msgr2=0x7f10540d1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:14.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.715+0000 7f105ba62700 1 --2- 172.21.15.113:0/3298574714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10540d1a50 0x7f10540d1ee0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f105409b2a0 tx=0x7f104c00b330 comp rx=0 tx=0).stop 2024-01-15T11:26:14.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.715+0000 7f105ba62700 1 -- 172.21.15.113:0/3298574714 shutdown_connections 2024-01-15T11:26:14.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.715+0000 7f105ba62700 1 --2- 172.21.15.113:0/3298574714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10540d1a50 0x7f10540d1ee0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.715+0000 7f105ba62700 1 --2- 172.21.15.113:0/3298574714 >> v2:172.21.15.141:3300/0 conn(0x7f105415bed0 0x7f10540b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.715+0000 7f105ba62700 1 --2- 172.21.15.113:0/3298574714 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10541603c0 0x7f10540d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.715+0000 7f105ba62700 1 -- 172.21.15.113:0/3298574714 >> 172.21.15.113:0/3298574714 conn(0x7f1054073130 msgr2=0x7f105409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:14.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.715+0000 7f105ba62700 1 -- 172.21.15.113:0/3298574714 shutdown_connections 2024-01-15T11:26:14.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.715+0000 7f105ba62700 1 -- 172.21.15.113:0/3298574714 wait complete. 2024-01-15T11:26:14.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.716+0000 7f105ba62700 1 Processor -- start 2024-01-15T11:26:14.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.716+0000 7f105ba62700 1 -- start start 2024-01-15T11:26:14.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f105ba62700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10540d1a50 0x7f10540b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:14.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f105ba62700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f105415bed0 0x7f105406c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:14.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f105ba62700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10541603c0 0x7f10540b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:14.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f105ba62700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10540b6320 con 0x7f105415bed0 2024-01-15T11:26:14.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f105ba62700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105406c4d0 con 0x7f10541603c0 2024-01-15T11:26:14.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f105ba62700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105406b900 con 0x7f10540d1a50 2024-01-15T11:26:14.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f1058ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f105415bed0 0x7f105406c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:14.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f1058ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f105415bed0 0x7f105406c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51956/0 (socket says 172.21.15.113:51956) 2024-01-15T11:26:14.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f1058ffd700 1 -- 172.21.15.113:0/4247475849 learned_addr learned my addr 172.21.15.113:0/4247475849 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f10597fe700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10540d1a50 0x7f10540b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f1059fff700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10541603c0 0x7f10540b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f1058ffd700 1 -- 172.21.15.113:0/4247475849 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10540d1a50 msgr2=0x7f10540b5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f1058ffd700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10540d1a50 0x7f10540b5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f1058ffd700 1 -- 172.21.15.113:0/4247475849 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10541603c0 msgr2=0x7f10540b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f1058ffd700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10541603c0 0x7f10540b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f1058ffd700 1 -- 172.21.15.113:0/4247475849 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c009b30 con 0x7f105415bed0 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.717+0000 7f1059fff700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10541603c0 0x7f10540b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.718+0000 7f1058ffd700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f105415bed0 0x7f105406c960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f104400d800 tx=0x7f104400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.718+0000 7f1049ffb700 1 -- 172.21.15.113:0/4247475849 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10440125f0 con 0x7f105415bed0 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.718+0000 7f1049ffb700 1 -- 172.21.15.113:0/4247475849 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1044012750 con 0x7f105415bed0 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.718+0000 7f1049ffb700 1 -- 172.21.15.113:0/4247475849 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044012b00 con 0x7f105415bed0 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.718+0000 7f105ba62700 1 -- 172.21.15.113:0/4247475849 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105406bba0 con 0x7f105415bed0 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.718+0000 7f105ba62700 1 -- 172.21.15.113:0/4247475849 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f105406c160 con 0x7f105415bed0 2024-01-15T11:26:14.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.719+0000 7f105ba62700 1 -- 172.21.15.113:0/4247475849 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f105406c680 con 0x7f105415bed0 2024-01-15T11:26:14.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.719+0000 7f1049ffb700 1 -- 172.21.15.113:0/4247475849 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1044012c60 con 0x7f105415bed0 2024-01-15T11:26:14.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.719+0000 7f1049ffb700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f103806b350 0x7f103806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:14.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.720+0000 7f1049ffb700 1 -- 172.21.15.113:0/4247475849 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1044089e50 con 0x7f105415bed0 2024-01-15T11:26:14.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.720+0000 7f10597fe700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f103806b350 0x7f103806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:14.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.720+0000 7f10597fe700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f103806b350 0x7f103806d810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1050009950 tx=0x7f1050008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:14.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.724+0000 7f1049ffb700 1 -- 172.21.15.113:0/4247475849 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f105406c680 con 0x7f105415bed0 2024-01-15T11:26:14.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.955+0000 7f105ba62700 1 -- 172.21.15.113:0/4247475849 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f105406c640 con 0x7f105415bed0 2024-01-15T11:26:14.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.955+0000 7f1049ffb700 1 -- 172.21.15.113:0/4247475849 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f1044054cc0 con 0x7f105415bed0 2024-01-15T11:26:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:14.955 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/1602579902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":1602579902},{"type":"v1","addr":"172.21.15.113:6836","nonce":1602579902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4915,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/3965637007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3965637007},{"type":"v1","addr":"172.21.15.141:6834","nonce":3965637007}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4918,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/4263133094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":4263133094},{"type":"v1","addr":"172.21.15.141:6835","nonce":4263133094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T11:26:10.938493+0000","modified":"2024-01-15T11:26:13.970629+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5004":{"gid":5004,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6837/436972884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":436972884},{"type":"v1","addr":"172.21.15.113:6837","nonce":436972884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-15T11:26:14.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.957+0000 7f105ba62700 1 -- 172.21.15.113:0/4247475849 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f103806b350 msgr2=0x7f103806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:14.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.957+0000 7f105ba62700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f103806b350 0x7f103806d810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1050009950 tx=0x7f1050008040 comp rx=0 tx=0).stop 2024-01-15T11:26:14.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.957+0000 7f105ba62700 1 -- 172.21.15.113:0/4247475849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f105415bed0 msgr2=0x7f105406c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:14.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.957+0000 7f105ba62700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f105415bed0 0x7f105406c960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f104400d800 tx=0x7f104400db10 comp rx=0 tx=0).stop 2024-01-15T11:26:14.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.957+0000 7f105ba62700 1 -- 172.21.15.113:0/4247475849 shutdown_connections 2024-01-15T11:26:14.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.957+0000 7f105ba62700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f103806b350 0x7f103806d810 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.957+0000 7f105ba62700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10541603c0 0x7f10540b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.957+0000 7f105ba62700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f10540d1a50 0x7f10540b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.958+0000 7f105ba62700 1 --2- 172.21.15.113:0/4247475849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f105415bed0 0x7f105406c960 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:14.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.958+0000 7f105ba62700 1 -- 172.21.15.113:0/4247475849 >> 172.21.15.113:0/4247475849 conn(0x7f1054073130 msgr2=0x7f1054159bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:14.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.958+0000 7f105ba62700 1 -- 172.21.15.113:0/4247475849 shutdown_connections 2024-01-15T11:26:14.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:14.958+0000 7f105ba62700 1 -- 172.21.15.113:0/4247475849 wait complete. 2024-01-15T11:26:14.958 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 17 2024-01-15T11:26:14.969 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:26:15.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.092+0000 7fa775866700 1 Processor -- start 2024-01-15T11:26:15.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.092+0000 7fa775866700 1 -- start start 2024-01-15T11:26:15.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.092+0000 7fa775866700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 0x7fa7700d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.092+0000 7fa775866700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa7700d3a00 0x7fa7700d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.092+0000 7fa775866700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7700d20b0 con 0x7fa7700d0db0 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.092+0000 7fa775866700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7700d3150 con 0x7fa7700d3a00 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.092+0000 7fa775866700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7700d1240 con 0x7fa77015e1e0 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.093+0000 7fa76e7fc700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fa7700d3a00 0x7fa7700d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.093+0000 7fa76effd700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 0x7fa7700d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.093+0000 7fa76effd700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 0x7fa7700d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:60122/0 (socket says 172.21.15.113:60122) 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.093+0000 7fa76effd700 1 -- 172.21.15.113:0/3478233138 learned_addr learned my addr 172.21.15.113:0/3478233138 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.093+0000 7fa76effd700 1 -- 172.21.15.113:0/3478233138 >> v2:172.21.15.141:3300/0 conn(0x7fa7700d3a00 msgr2=0x7fa7700d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.093+0000 7fa76effd700 1 --2- 172.21.15.113:0/3478233138 >> v2:172.21.15.141:3300/0 conn(0x7fa7700d3a00 0x7fa7700d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.093+0000 7fa76effd700 1 -- 172.21.15.113:0/3478233138 >> v1:172.21.15.141:6789/0 conn(0x7fa7700d0db0 legacy=0x7fa77006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.093+0000 7fa76effd700 1 -- 172.21.15.113:0/3478233138 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7700d2390 con 0x7fa77015e1e0 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.093+0000 7fa76effd700 1 --2- 172.21.15.113:0/3478233138 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 0x7fa7700d34c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa7600098f0 tx=0x7fa760009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34be0de6000d6fe2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.094+0000 7fa76dffb700 1 -- 172.21.15.113:0/3478233138 <== mon.2 v2:172.21.15.141:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760011830 con 0x7fa77015e1e0 2024-01-15T11:26:15.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.094+0000 7fa76dffb700 1 -- 172.21.15.113:0/3478233138 <== mon.2 v2:172.21.15.141:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa760011990 con 0x7fa77015e1e0 2024-01-15T11:26:15.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.094+0000 7fa76dffb700 1 -- 172.21.15.113:0/3478233138 <== mon.2 v2:172.21.15.141:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760011d20 con 0x7fa77015e1e0 2024-01-15T11:26:15.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.094+0000 7fa775866700 1 -- 172.21.15.113:0/3478233138 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 msgr2=0x7fa7700d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.094+0000 7fa775866700 1 --2- 172.21.15.113:0/3478233138 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 0x7fa7700d34c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa7600098f0 tx=0x7fa760009c00 comp rx=0 tx=0).stop 2024-01-15T11:26:15.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.094+0000 7fa775866700 1 -- 172.21.15.113:0/3478233138 shutdown_connections 2024-01-15T11:26:15.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.094+0000 7fa775866700 1 --2- 172.21.15.113:0/3478233138 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 0x7fa7700d34c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.094+0000 7fa775866700 1 --2- 172.21.15.113:0/3478233138 >> v2:172.21.15.141:3300/0 conn(0x7fa7700d3a00 0x7fa7700d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.094+0000 7fa775866700 1 -- 172.21.15.113:0/3478233138 >> 172.21.15.113:0/3478233138 conn(0x7fa7700b5b50 msgr2=0x7fa7700b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:15.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.094+0000 7fa775866700 1 -- 172.21.15.113:0/3478233138 shutdown_connections 2024-01-15T11:26:15.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.094+0000 7fa775866700 1 -- 172.21.15.113:0/3478233138 wait complete. 2024-01-15T11:26:15.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.095+0000 7fa775866700 1 Processor -- start 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.096+0000 7fa775866700 1 -- start start 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.096+0000 7fa775866700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7700d0db0 0x7fa7700cb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.096+0000 7fa775866700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7700d3a00 0x7fa7700cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.096+0000 7fa775866700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 0x7fa7700b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.096+0000 7fa775866700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7700cd1d0 con 0x7fa7700d3a00 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.096+0000 7fa775866700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7700cc2d0 con 0x7fa7700d0db0 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.096+0000 7fa775866700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7700cc550 con 0x7fa77015e1e0 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.096+0000 7fa76e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7700d3a00 0x7fa7700cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.096+0000 7fa76e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7700d3a00 0x7fa7700cbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51970/0 (socket says 172.21.15.113:51970) 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.096+0000 7fa76e7fc700 1 -- 172.21.15.113:0/707503438 learned_addr learned my addr 172.21.15.113:0/707503438 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa76effd700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7700d0db0 0x7fa7700cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa76f7fe700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 0x7fa7700b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa76e7fc700 1 -- 172.21.15.113:0/707503438 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 msgr2=0x7fa7700b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa76e7fc700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 0x7fa7700b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa76e7fc700 1 -- 172.21.15.113:0/707503438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7700d0db0 msgr2=0x7fa7700cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa76e7fc700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7700d0db0 0x7fa7700cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa76e7fc700 1 -- 172.21.15.113:0/707503438 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa760009550 con 0x7fa7700d3a00 2024-01-15T11:26:15.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa76f7fe700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 0x7fa7700b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:15.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa76e7fc700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7700d3a00 0x7fa7700cbd90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa75800e9b0 tx=0x7fa75800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:15.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa767fff700 1 -- 172.21.15.113:0/707503438 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758009710 con 0x7fa7700d3a00 2024-01-15T11:26:15.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa767fff700 1 -- 172.21.15.113:0/707503438 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75800c850 con 0x7fa7700d3a00 2024-01-15T11:26:15.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa775866700 1 -- 172.21.15.113:0/707503438 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7700cc830 con 0x7fa7700d3a00 2024-01-15T11:26:15.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa767fff700 1 -- 172.21.15.113:0/707503438 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75800cbc0 con 0x7fa7700d3a00 2024-01-15T11:26:15.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.097+0000 7fa775866700 1 -- 172.21.15.113:0/707503438 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7700ccdf0 con 0x7fa7700d3a00 2024-01-15T11:26:15.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.098+0000 7fa775866700 1 -- 172.21.15.113:0/707503438 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7700d4d10 con 0x7fa7700d3a00 2024-01-15T11:26:15.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.098+0000 7fa767fff700 1 -- 172.21.15.113:0/707503438 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa75800cd20 con 0x7fa7700d3a00 2024-01-15T11:26:15.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.099+0000 7fa767fff700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa75006b350 0x7fa75006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.099+0000 7fa767fff700 1 -- 172.21.15.113:0/707503438 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa758089c60 con 0x7fa7700d3a00 2024-01-15T11:26:15.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.099+0000 7fa76effd700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa75006b350 0x7fa75006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.099+0000 7fa76effd700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa75006b350 0x7fa75006d810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa760000c00 tx=0x7fa760008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:15.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.103+0000 7fa767fff700 1 -- 172.21.15.113:0/707503438 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa758054b60 con 0x7fa7700d3a00 2024-01-15T11:26:15.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.295+0000 7fa775866700 1 -- 172.21.15.113:0/707503438 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7700751e0 con 0x7fa7700d3a00 2024-01-15T11:26:15.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.296+0000 7fa767fff700 1 -- 172.21.15.113:0/707503438 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16629 (secure 0 0 0) 0x7fa758021090 con 0x7fa7700d3a00 2024-01-15T11:26:15.296 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:15.296 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":33,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:26:12.537199+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-15T11:26:08.840011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-15T11:26:09.908268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:26:15.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.298+0000 7fa775866700 1 -- 172.21.15.113:0/707503438 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa75006b350 msgr2=0x7fa75006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.298+0000 7fa775866700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa75006b350 0x7fa75006d810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa760000c00 tx=0x7fa760008030 comp rx=0 tx=0).stop 2024-01-15T11:26:15.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.298+0000 7fa775866700 1 -- 172.21.15.113:0/707503438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7700d3a00 msgr2=0x7fa7700cbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.298+0000 7fa775866700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7700d3a00 0x7fa7700cbd90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa75800e9b0 tx=0x7fa75800ed70 comp rx=0 tx=0).stop 2024-01-15T11:26:15.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.299+0000 7fa775866700 1 -- 172.21.15.113:0/707503438 shutdown_connections 2024-01-15T11:26:15.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.299+0000 7fa775866700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa75006b350 0x7fa75006d810 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.299+0000 7fa775866700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa77015e1e0 0x7fa7700b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.299+0000 7fa775866700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa7700d0db0 0x7fa7700cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.299+0000 7fa775866700 1 --2- 172.21.15.113:0/707503438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7700d3a00 0x7fa7700cbd90 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.299+0000 7fa775866700 1 -- 172.21.15.113:0/707503438 >> 172.21.15.113:0/707503438 conn(0x7fa7700b5b50 msgr2=0x7fa7700b7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:15.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.299+0000 7fa775866700 1 -- 172.21.15.113:0/707503438 shutdown_connections 2024-01-15T11:26:15.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.299+0000 7fa775866700 1 -- 172.21.15.113:0/707503438 wait complete. 2024-01-15T11:26:15.310 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T11:26:15.310 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T11:26:15.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.431+0000 7fc506062700 1 Processor -- start 2024-01-15T11:26:15.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.431+0000 7fc506062700 1 -- start start 2024-01-15T11:26:15.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.432+0000 7fc506062700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5000b4350 0x7fc5000af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.432+0000 7fc506062700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5000b0290 con 0x7fc5000b5dd0 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.432+0000 7fc506062700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5000b4820 con 0x7fc5000b59f0 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.432+0000 7fc506062700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5000b4b00 con 0x7fc5000b4350 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.432+0000 7fc4feffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fc5000b5dd0 0x7fc5000b3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56456/0 (socket says 172.21.15.113:56456) 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.432+0000 7fc4feffd700 1 -- 172.21.15.113:0/3941686981 learned_addr learned my addr 172.21.15.113:0/3941686981 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735801212 0 0) 0x7fc5000b0290 con 0x7fc5000b5dd0 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4e80035b0 con 0x7fc5000b5dd0 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fffff700 1 --2- 172.21.15.113:0/3941686981 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5000b4350 0x7fc5000af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 816189725 0 0) 0x7fc4e80035b0 con 0x7fc5000b5dd0 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5000b4350 msgr2=0x7fc5000af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 --2- 172.21.15.113:0/3941686981 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5000b4350 0x7fc5000af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 >> v1:172.21.15.141:6789/0 conn(0x7fc5000b59f0 legacy=0x7fc50006db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5000ae9b0 con 0x7fc5000b5dd0 2024-01-15T11:26:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fc4f4001670 con 0x7fc5000b5dd0 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 >> v1:172.21.15.113:6789/0 conn(0x7fc5000b5dd0 legacy=0x7fc5000b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 --2- 172.21.15.113:0/3941686981 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc4e8003a30 0x7fc4e8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 --2- 172.21.15.113:0/3941686981 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5000b4350 0x7fc4e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 --2- 172.21.15.113:0/3941686981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4e8004d90 0x7fc4e8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5000b4b00 con 0x7fc4e8004d90 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e800bba0 con 0x7fc4e8003a30 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e800bd30 con 0x7fc5000b4350 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fffff700 1 --2- 172.21.15.113:0/3941686981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4e8004d90 0x7fc4e8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4ff7fe700 1 --2- 172.21.15.113:0/3941686981 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc4e8003a30 0x7fc4e8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4feffd700 1 --2- 172.21.15.113:0/3941686981 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5000b4350 0x7fc4e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fffff700 1 -- 172.21.15.113:0/3941686981 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5000b4350 msgr2=0x7fc4e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.433+0000 7fc4fffff700 1 --2- 172.21.15.113:0/3941686981 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc5000b4350 0x7fc4e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc4fffff700 1 -- 172.21.15.113:0/3941686981 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc4e8003a30 msgr2=0x7fc4e8003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:15.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc4fffff700 1 --2- 172.21.15.113:0/3941686981 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc4e8003a30 0x7fc4e8003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc4fffff700 1 -- 172.21.15.113:0/3941686981 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e80035b0 con 0x7fc4e8004d90 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc4fffff700 1 --2- 172.21.15.113:0/3941686981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4e8004d90 0x7fc4e8005210 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8007360 tx=0x7fc4e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6279aaa8b9813b76 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8007e80 con 0x7fc4e8004d90 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f8014400 con 0x7fc4e8004d90 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc4fe7fc700 1 -- 172.21.15.113:0/3941686981 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8014790 con 0x7fc4e8004d90 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc506062700 1 -- 172.21.15.113:0/3941686981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4e8004d90 msgr2=0x7fc4e8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc506062700 1 --2- 172.21.15.113:0/3941686981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4e8004d90 0x7fc4e8005210 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8007360 tx=0x7fc4e8005880 comp rx=0 tx=0).stop 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc4fffff700 1 -- 172.21.15.113:0/3941686981 reap_dead start 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc506062700 1 -- 172.21.15.113:0/3941686981 shutdown_connections 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc506062700 1 -- 172.21.15.113:0/3941686981 >> 172.21.15.113:0/3941686981 conn(0x7fc500072990 msgr2=0x7fc50006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc506062700 1 -- 172.21.15.113:0/3941686981 shutdown_connections 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.434+0000 7fc506062700 1 -- 172.21.15.113:0/3941686981 wait complete. 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.436+0000 7fc506062700 1 Processor -- start 2024-01-15T11:26:15.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.436+0000 7fc506062700 1 -- start start 2024-01-15T11:26:15.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.436+0000 7fc506062700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc50014e050 0x7fc50014fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.436+0000 7fc506062700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc500150380 0x7fc500160d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.436+0000 7fc506062700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc50014e7a0 0x7fc500161290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.436+0000 7fc506062700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc500151bb0 con 0x7fc50014e7a0 2024-01-15T11:26:15.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc506062700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc500151e90 con 0x7fc500150380 2024-01-15T11:26:15.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc506062700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc500152170 con 0x7fc50014e050 2024-01-15T11:26:15.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4fffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc50014e7a0 0x7fc500161290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4fffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc50014e7a0 0x7fc500161290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51996/0 (socket says 172.21.15.113:51996) 2024-01-15T11:26:15.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4fffff700 1 -- 172.21.15.113:0/532686605 learned_addr learned my addr 172.21.15.113:0/532686605 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:15.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4ff7fe700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc50014e050 0x7fc50014fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4feffd700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc500150380 0x7fc500160d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4fffff700 1 -- 172.21.15.113:0/532686605 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc50014e050 msgr2=0x7fc50014fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4fffff700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc50014e050 0x7fc50014fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4fffff700 1 -- 172.21.15.113:0/532686605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc500150380 msgr2=0x7fc500160d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4fffff700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc500150380 0x7fc500160d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4fffff700 1 -- 172.21.15.113:0/532686605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc500152450 con 0x7fc50014e7a0 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4fffff700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc50014e7a0 0x7fc500161290 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8014000 tx=0x7fc4f8004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4e7fff700 1 -- 172.21.15.113:0/532686605 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8007000 con 0x7fc50014e7a0 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4e7fff700 1 -- 172.21.15.113:0/532686605 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f801e440 con 0x7fc50014e7a0 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc506062700 1 -- 172.21.15.113:0/532686605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc50014e430 con 0x7fc50014e7a0 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc4e7fff700 1 -- 172.21.15.113:0/532686605 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8009410 con 0x7fc50014e7a0 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.437+0000 7fc506062700 1 -- 172.21.15.113:0/532686605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5000ce2f0 con 0x7fc50014e7a0 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.438+0000 7fc506062700 1 -- 172.21.15.113:0/532686605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc500067f60 con 0x7fc50014e7a0 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.439+0000 7fc4e7fff700 1 -- 172.21.15.113:0/532686605 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc4f801e7f0 con 0x7fc50014e7a0 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.439+0000 7fc4e7fff700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc4ec06b350 0x7fc4ec06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.439+0000 7fc4e7fff700 1 -- 172.21.15.113:0/532686605 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4f808af10 con 0x7fc50014e7a0 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.439+0000 7fc4ff7fe700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc4ec06b350 0x7fc4ec06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.439+0000 7fc4ff7fe700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc4ec06b350 0x7fc4ec06d810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0007fd0 tx=0x7fc4f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:15.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.444+0000 7fc4e7fff700 1 -- 172.21.15.113:0/532686605 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4f8055ed0 con 0x7fc50014e7a0 2024-01-15T11:26:15.519 INFO:tasks.ceph.mds.a.smithi113.stderr:2024-01-15T11:26:15.520+0000 7f3112210700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T11:26:15.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.634+0000 7fc506062700 1 -- 172.21.15.113:0/532686605 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc50006e280 con 0x7fc4ec06b350 2024-01-15T11:26:15.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.643+0000 7fc4e7fff700 1 -- 172.21.15.113:0/532686605 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289275 (secure 0 0 0) 0x7fc4f0006860 con 0x7fc4ec06b350 2024-01-15T11:26:15.646 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:15.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.646+0000 7fc506062700 1 -- 172.21.15.113:0/532686605 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc4ec06b350 msgr2=0x7fc4ec06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.646+0000 7fc506062700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc4ec06b350 0x7fc4ec06d810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0007fd0 tx=0x7fc4f0009040 comp rx=0 tx=0).stop 2024-01-15T11:26:15.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.646+0000 7fc506062700 1 -- 172.21.15.113:0/532686605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc50014e7a0 msgr2=0x7fc500161290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.646+0000 7fc506062700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc50014e7a0 0x7fc500161290 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8014000 tx=0x7fc4f8004b40 comp rx=0 tx=0).stop 2024-01-15T11:26:15.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.646+0000 7fc506062700 1 -- 172.21.15.113:0/532686605 shutdown_connections 2024-01-15T11:26:15.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.646+0000 7fc506062700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc4ec06b350 0x7fc4ec06d810 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.646+0000 7fc506062700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc50014e7a0 0x7fc500161290 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.646+0000 7fc506062700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc50014e050 0x7fc50014fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.646+0000 7fc506062700 1 --2- 172.21.15.113:0/532686605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc500150380 0x7fc500160d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.646+0000 7fc506062700 1 -- 172.21.15.113:0/532686605 >> 172.21.15.113:0/532686605 conn(0x7fc500072990 msgr2=0x7fc50006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:15.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.647+0000 7fc506062700 1 -- 172.21.15.113:0/532686605 shutdown_connections 2024-01-15T11:26:15.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.647+0000 7fc506062700 1 -- 172.21.15.113:0/532686605 wait complete. 2024-01-15T11:26:15.650 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-15T11:26:15.658 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-01-15T11:26:14.120651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220144,"kb_used_data":3948,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754576,"statfs":{"total":773094113280,"available":772868685824,"internally_reserved":0,"allocated":4042752,"data_stored":2271487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.588869"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.567900+0000","last_change":"2024-01-15T11:26:11.564123+0000","last_active":"2024-01-15T11:26:11.567900+0000","last_peered":"2024-01-15T11:26:11.567900+0000","last_clean":"2024-01-15T11:26:11.567900+0000","last_became_active":"2024-01-15T11:26:11.563850+0000","last_became_peered":"2024-01-15T11:26:11.563850+0000","last_unstale":"2024-01-15T11:26:11.567900+0000","last_undegraded":"2024-01-15T11:26:11.567900+0000","last_fullsized":"2024-01-15T11:26:11.567900+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:39.545648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560581+0000","last_change":"2024-01-15T11:26:11.555663+0000","last_active":"2024-01-15T11:26:11.560581+0000","last_peered":"2024-01-15T11:26:11.560581+0000","last_clean":"2024-01-15T11:26:11.560581+0000","last_became_active":"2024-01-15T11:26:11.550879+0000","last_became_peered":"2024-01-15T11:26:11.550879+0000","last_unstale":"2024-01-15T11:26:11.560581+0000","last_undegraded":"2024-01-15T11:26:11.560581+0000","last_fullsized":"2024-01-15T11:26:11.560581+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:40:49.107290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.568044+0000","last_change":"2024-01-15T11:26:11.566884+0000","last_active":"2024-01-15T11:26:11.568044+0000","last_peered":"2024-01-15T11:26:11.568044+0000","last_clean":"2024-01-15T11:26:11.568044+0000","last_became_active":"2024-01-15T11:26:11.566612+0000","last_became_peered":"2024-01-15T11:26:11.566612+0000","last_unstale":"2024-01-15T11:26:11.568044+0000","last_undegraded":"2024-01-15T11:26:11.568044+0000","last_fullsized":"2024-01-15T11:26:11.568044+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:12.731404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.569507+0000","last_change":"2024-01-15T11:26:11.567887+0000","last_active":"2024-01-15T11:26:11.569507+0000","last_peered":"2024-01-15T11:26:11.569507+0000","last_clean":"2024-01-15T11:26:11.569507+0000","last_became_active":"2024-01-15T11:26:11.567624+0000","last_became_peered":"2024-01-15T11:26:11.567624+0000","last_unstale":"2024-01-15T11:26:11.569507+0000","last_undegraded":"2024-01-15T11:26:11.569507+0000","last_fullsized":"2024-01-15T11:26:11.569507+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:27:57.735875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565135+0000","last_change":"2024-01-15T11:26:11.561906+0000","last_active":"2024-01-15T11:26:11.565135+0000","last_peered":"2024-01-15T11:26:11.565135+0000","last_clean":"2024-01-15T11:26:11.565135+0000","last_became_active":"2024-01-15T11:26:11.561631+0000","last_became_peered":"2024-01-15T11:26:11.561631+0000","last_unstale":"2024-01-15T11:26:11.565135+0000","last_undegraded":"2024-01-15T11:26:11.565135+0000","last_fullsized":"2024-01-15T11:26:11.565135+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:47:02.390456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560691+0000","last_change":"2024-01-15T11:26:11.553208+0000","last_active":"2024-01-15T11:26:11.560691+0000","last_peered":"2024-01-15T11:26:11.560691+0000","last_clean":"2024-01-15T11:26:11.560691+0000","last_became_active":"2024-01-15T11:26:11.550184+0000","last_became_peered":"2024-01-15T11:26:11.550184+0000","last_unstale":"2024-01-15T11:26:11.560691+0000","last_undegraded":"2024-01-15T11:26:11.560691+0000","last_fullsized":"2024-01-15T11:26:11.560691+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:59:32.790611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564334+0000","last_change":"2024-01-15T11:26:11.562018+0000","last_active":"2024-01-15T11:26:11.564334+0000","last_peered":"2024-01-15T11:26:11.564334+0000","last_clean":"2024-01-15T11:26:11.564334+0000","last_became_active":"2024-01-15T11:26:11.561589+0000","last_became_peered":"2024-01-15T11:26:11.561589+0000","last_unstale":"2024-01-15T11:26:11.564334+0000","last_undegraded":"2024-01-15T11:26:11.564334+0000","last_fullsized":"2024-01-15T11:26:11.564334+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:09:11.710169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.563345+0000","last_change":"2024-01-15T11:26:11.559776+0000","last_active":"2024-01-15T11:26:11.563345+0000","last_peered":"2024-01-15T11:26:11.563345+0000","last_clean":"2024-01-15T11:26:11.563345+0000","last_became_active":"2024-01-15T11:26:11.559428+0000","last_became_peered":"2024-01-15T11:26:11.559428+0000","last_unstale":"2024-01-15T11:26:11.563345+0000","last_undegraded":"2024-01-15T11:26:11.563345+0000","last_fullsized":"2024-01-15T11:26:11.563345+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:54:28.575899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.569858+0000","last_change":"2024-01-15T11:26:11.568633+0000","last_active":"2024-01-15T11:26:11.569858+0000","last_peered":"2024-01-15T11:26:11.569858+0000","last_clean":"2024-01-15T11:26:11.569858+0000","last_became_active":"2024-01-15T11:26:11.568362+0000","last_became_peered":"2024-01-15T11:26:11.568362+0000","last_unstale":"2024-01-15T11:26:11.569858+0000","last_undegraded":"2024-01-15T11:26:11.569858+0000","last_fullsized":"2024-01-15T11:26:11.569858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:35:16.780032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566021+0000","last_change":"2024-01-15T11:26:11.561718+0000","last_active":"2024-01-15T11:26:11.566021+0000","last_peered":"2024-01-15T11:26:11.566021+0000","last_clean":"2024-01-15T11:26:11.566021+0000","last_became_active":"2024-01-15T11:26:11.561465+0000","last_became_peered":"2024-01-15T11:26:11.561465+0000","last_unstale":"2024-01-15T11:26:11.566021+0000","last_undegraded":"2024-01-15T11:26:11.566021+0000","last_fullsized":"2024-01-15T11:26:11.566021+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:39.545648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.568328+0000","last_change":"2024-01-15T11:26:11.565927+0000","last_active":"2024-01-15T11:26:11.568328+0000","last_peered":"2024-01-15T11:26:11.568328+0000","last_clean":"2024-01-15T11:26:11.568328+0000","last_became_active":"2024-01-15T11:26:11.565645+0000","last_became_peered":"2024-01-15T11:26:11.565645+0000","last_unstale":"2024-01-15T11:26:11.568328+0000","last_undegraded":"2024-01-15T11:26:11.568328+0000","last_fullsized":"2024-01-15T11:26:11.568328+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:12:09.208938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.551539+0000","last_change":"2024-01-15T11:26:11.546800+0000","last_active":"2024-01-15T11:26:11.551539+0000","last_peered":"2024-01-15T11:26:11.551539+0000","last_clean":"2024-01-15T11:26:11.551539+0000","last_became_active":"2024-01-15T11:26:11.546487+0000","last_became_peered":"2024-01-15T11:26:11.546487+0000","last_unstale":"2024-01-15T11:26:11.551539+0000","last_undegraded":"2024-01-15T11:26:11.551539+0000","last_fullsized":"2024-01-15T11:26:11.551539+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:52.795135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565154+0000","last_change":"2024-01-15T11:26:11.562595+0000","last_active":"2024-01-15T11:26:11.565154+0000","last_peered":"2024-01-15T11:26:11.565154+0000","last_clean":"2024-01-15T11:26:11.565154+0000","last_became_active":"2024-01-15T11:26:11.562326+0000","last_became_peered":"2024-01-15T11:26:11.562326+0000","last_unstale":"2024-01-15T11:26:11.565154+0000","last_undegraded":"2024-01-15T11:26:11.565154+0000","last_fullsized":"2024-01-15T11:26:11.565154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:38:46.949689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560735+0000","last_change":"2024-01-15T11:26:11.559121+0000","last_active":"2024-01-15T11:26:11.560735+0000","last_peered":"2024-01-15T11:26:11.560735+0000","last_clean":"2024-01-15T11:26:11.560735+0000","last_became_active":"2024-01-15T11:26:11.558880+0000","last_became_peered":"2024-01-15T11:26:11.558880+0000","last_unstale":"2024-01-15T11:26:11.560735+0000","last_undegraded":"2024-01-15T11:26:11.560735+0000","last_fullsized":"2024-01-15T11:26:11.560735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:51:14.891621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.563693+0000","last_change":"2024-01-15T11:26:11.560914+0000","last_active":"2024-01-15T11:26:11.563693+0000","last_peered":"2024-01-15T11:26:11.563693+0000","last_clean":"2024-01-15T11:26:11.563693+0000","last_became_active":"2024-01-15T11:26:11.560577+0000","last_became_peered":"2024-01-15T11:26:11.560577+0000","last_unstale":"2024-01-15T11:26:11.563693+0000","last_undegraded":"2024-01-15T11:26:11.563693+0000","last_fullsized":"2024-01-15T11:26:11.563693+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:26.983127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564293+0000","last_change":"2024-01-15T11:26:11.560881+0000","last_active":"2024-01-15T11:26:11.564293+0000","last_peered":"2024-01-15T11:26:11.564293+0000","last_clean":"2024-01-15T11:26:11.564293+0000","last_became_active":"2024-01-15T11:26:11.560569+0000","last_became_peered":"2024-01-15T11:26:11.560569+0000","last_unstale":"2024-01-15T11:26:11.564293+0000","last_undegraded":"2024-01-15T11:26:11.564293+0000","last_fullsized":"2024-01-15T11:26:11.564293+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:16.794640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.563321+0000","last_change":"2024-01-15T11:26:11.559811+0000","last_active":"2024-01-15T11:26:11.563321+0000","last_peered":"2024-01-15T11:26:11.563321+0000","last_clean":"2024-01-15T11:26:11.563321+0000","last_became_active":"2024-01-15T11:26:11.555585+0000","last_became_peered":"2024-01-15T11:26:11.555585+0000","last_unstale":"2024-01-15T11:26:11.563321+0000","last_undegraded":"2024-01-15T11:26:11.563321+0000","last_fullsized":"2024-01-15T11:26:11.563321+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:40.189089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.570449+0000","last_change":"2024-01-15T11:26:11.563623+0000","last_active":"2024-01-15T11:26:11.570449+0000","last_peered":"2024-01-15T11:26:11.570449+0000","last_clean":"2024-01-15T11:26:11.570449+0000","last_became_active":"2024-01-15T11:26:11.563335+0000","last_became_peered":"2024-01-15T11:26:11.563335+0000","last_unstale":"2024-01-15T11:26:11.570449+0000","last_undegraded":"2024-01-15T11:26:11.570449+0000","last_fullsized":"2024-01-15T11:26:11.570449+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:54:28.575899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.571068+0000","last_change":"2024-01-15T11:26:11.568691+0000","last_active":"2024-01-15T11:26:11.571068+0000","last_peered":"2024-01-15T11:26:11.571068+0000","last_clean":"2024-01-15T11:26:11.571068+0000","last_became_active":"2024-01-15T11:26:11.568396+0000","last_became_peered":"2024-01-15T11:26:11.568396+0000","last_unstale":"2024-01-15T11:26:11.571068+0000","last_undegraded":"2024-01-15T11:26:11.571068+0000","last_fullsized":"2024-01-15T11:26:11.571068+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:40.189089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564924+0000","last_change":"2024-01-15T11:26:11.558062+0000","last_active":"2024-01-15T11:26:11.564924+0000","last_peered":"2024-01-15T11:26:11.564924+0000","last_clean":"2024-01-15T11:26:11.564924+0000","last_became_active":"2024-01-15T11:26:11.557802+0000","last_became_peered":"2024-01-15T11:26:11.557802+0000","last_unstale":"2024-01-15T11:26:11.564924+0000","last_undegraded":"2024-01-15T11:26:11.564924+0000","last_fullsized":"2024-01-15T11:26:11.564924+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:33:18.117063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566567+0000","last_change":"2024-01-15T11:26:11.562143+0000","last_active":"2024-01-15T11:26:11.566567+0000","last_peered":"2024-01-15T11:26:11.566567+0000","last_clean":"2024-01-15T11:26:11.566567+0000","last_became_active":"2024-01-15T11:26:11.561707+0000","last_became_peered":"2024-01-15T11:26:11.561707+0000","last_unstale":"2024-01-15T11:26:11.566567+0000","last_undegraded":"2024-01-15T11:26:11.566567+0000","last_fullsized":"2024-01-15T11:26:11.566567+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:27:57.735875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566565+0000","last_change":"2024-01-15T11:26:11.562532+0000","last_active":"2024-01-15T11:26:11.566565+0000","last_peered":"2024-01-15T11:26:11.566565+0000","last_clean":"2024-01-15T11:26:11.566565+0000","last_became_active":"2024-01-15T11:26:11.562246+0000","last_became_peered":"2024-01-15T11:26:11.562246+0000","last_unstale":"2024-01-15T11:26:11.566565+0000","last_undegraded":"2024-01-15T11:26:11.566565+0000","last_fullsized":"2024-01-15T11:26:11.566565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:32:06.812304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.561304+0000","last_change":"2024-01-15T11:26:11.555289+0000","last_active":"2024-01-15T11:26:11.561304+0000","last_peered":"2024-01-15T11:26:11.561304+0000","last_clean":"2024-01-15T11:26:11.561304+0000","last_became_active":"2024-01-15T11:26:11.554832+0000","last_became_peered":"2024-01-15T11:26:11.554832+0000","last_unstale":"2024-01-15T11:26:11.561304+0000","last_undegraded":"2024-01-15T11:26:11.561304+0000","last_fullsized":"2024-01-15T11:26:11.561304+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:18.187321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.562549+0000","last_change":"2024-01-15T11:26:11.557553+0000","last_active":"2024-01-15T11:26:11.562549+0000","last_peered":"2024-01-15T11:26:11.562549+0000","last_clean":"2024-01-15T11:26:11.562549+0000","last_became_active":"2024-01-15T11:26:11.557206+0000","last_became_peered":"2024-01-15T11:26:11.557206+0000","last_unstale":"2024-01-15T11:26:11.562549+0000","last_undegraded":"2024-01-15T11:26:11.562549+0000","last_fullsized":"2024-01-15T11:26:11.562549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:09:11.710169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.567761+0000","last_change":"2024-01-15T11:26:11.560669+0000","last_active":"2024-01-15T11:26:11.567761+0000","last_peered":"2024-01-15T11:26:11.567761+0000","last_clean":"2024-01-15T11:26:11.567761+0000","last_became_active":"2024-01-15T11:26:11.560364+0000","last_became_peered":"2024-01-15T11:26:11.560364+0000","last_unstale":"2024-01-15T11:26:11.567761+0000","last_undegraded":"2024-01-15T11:26:11.567761+0000","last_fullsized":"2024-01-15T11:26:11.567761+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:04:01.689068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560610+0000","last_change":"2024-01-15T11:26:11.557494+0000","last_active":"2024-01-15T11:26:11.560610+0000","last_peered":"2024-01-15T11:26:11.560610+0000","last_clean":"2024-01-15T11:26:11.560610+0000","last_became_active":"2024-01-15T11:26:11.555709+0000","last_became_peered":"2024-01-15T11:26:11.555709+0000","last_unstale":"2024-01-15T11:26:11.560610+0000","last_undegraded":"2024-01-15T11:26:11.560610+0000","last_fullsized":"2024-01-15T11:26:11.560610+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:26.983127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.569547+0000","last_change":"2024-01-15T11:26:11.568325+0000","last_active":"2024-01-15T11:26:11.569547+0000","last_peered":"2024-01-15T11:26:11.569547+0000","last_clean":"2024-01-15T11:26:11.569547+0000","last_became_active":"2024-01-15T11:26:11.567570+0000","last_became_peered":"2024-01-15T11:26:11.567570+0000","last_unstale":"2024-01-15T11:26:11.569547+0000","last_undegraded":"2024-01-15T11:26:11.569547+0000","last_fullsized":"2024-01-15T11:26:11.569547+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:23:17.014644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.554443+0000","last_change":"2024-01-15T11:26:11.550678+0000","last_active":"2024-01-15T11:26:11.554443+0000","last_peered":"2024-01-15T11:26:11.554443+0000","last_clean":"2024-01-15T11:26:11.554443+0000","last_became_active":"2024-01-15T11:26:11.550389+0000","last_became_peered":"2024-01-15T11:26:11.550389+0000","last_unstale":"2024-01-15T11:26:11.554443+0000","last_undegraded":"2024-01-15T11:26:11.554443+0000","last_fullsized":"2024-01-15T11:26:11.554443+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:40.189089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566581+0000","last_change":"2024-01-15T11:26:11.562364+0000","last_active":"2024-01-15T11:26:11.566581+0000","last_peered":"2024-01-15T11:26:11.566581+0000","last_clean":"2024-01-15T11:26:11.566581+0000","last_became_active":"2024-01-15T11:26:11.562012+0000","last_became_peered":"2024-01-15T11:26:11.562012+0000","last_unstale":"2024-01-15T11:26:11.566581+0000","last_undegraded":"2024-01-15T11:26:11.566581+0000","last_fullsized":"2024-01-15T11:26:11.566581+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:33:35.657813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564263+0000","last_change":"2024-01-15T11:26:11.560923+0000","last_active":"2024-01-15T11:26:11.564263+0000","last_peered":"2024-01-15T11:26:11.564263+0000","last_clean":"2024-01-15T11:26:11.564263+0000","last_became_active":"2024-01-15T11:26:11.560655+0000","last_became_peered":"2024-01-15T11:26:11.560655+0000","last_unstale":"2024-01-15T11:26:11.564263+0000","last_undegraded":"2024-01-15T11:26:11.564263+0000","last_fullsized":"2024-01-15T11:26:11.564263+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:12.731404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.567637+0000","last_change":"2024-01-15T11:26:11.559945+0000","last_active":"2024-01-15T11:26:11.567637+0000","last_peered":"2024-01-15T11:26:11.567637+0000","last_clean":"2024-01-15T11:26:11.567637+0000","last_became_active":"2024-01-15T11:26:11.559320+0000","last_became_peered":"2024-01-15T11:26:11.559320+0000","last_unstale":"2024-01-15T11:26:11.567637+0000","last_undegraded":"2024-01-15T11:26:11.567637+0000","last_fullsized":"2024-01-15T11:26:11.567637+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:32:43.889552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.559627+0000","last_change":"2024-01-15T11:26:11.557039+0000","last_active":"2024-01-15T11:26:11.559627+0000","last_peered":"2024-01-15T11:26:11.559627+0000","last_clean":"2024-01-15T11:26:11.559627+0000","last_became_active":"2024-01-15T11:26:11.555633+0000","last_became_peered":"2024-01-15T11:26:11.555633+0000","last_unstale":"2024-01-15T11:26:11.559627+0000","last_undegraded":"2024-01-15T11:26:11.559627+0000","last_fullsized":"2024-01-15T11:26:11.559627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:02:39.352286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.562212+0000","last_change":"2024-01-15T11:26:11.555082+0000","last_active":"2024-01-15T11:26:11.562212+0000","last_peered":"2024-01-15T11:26:11.562212+0000","last_clean":"2024-01-15T11:26:11.562212+0000","last_became_active":"2024-01-15T11:26:11.554758+0000","last_became_peered":"2024-01-15T11:26:11.554758+0000","last_unstale":"2024-01-15T11:26:11.562212+0000","last_undegraded":"2024-01-15T11:26:11.562212+0000","last_fullsized":"2024-01-15T11:26:11.562212+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:14:25.674690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565225+0000","last_change":"2024-01-15T11:26:10.551768+0000","last_active":"2024-01-15T11:26:11.565225+0000","last_peered":"2024-01-15T11:26:11.565225+0000","last_clean":"2024-01-15T11:26:11.565225+0000","last_became_active":"2024-01-15T11:26:10.551469+0000","last_became_peered":"2024-01-15T11:26:10.551469+0000","last_unstale":"2024-01-15T11:26:11.565225+0000","last_undegraded":"2024-01-15T11:26:11.565225+0000","last_fullsized":"2024-01-15T11:26:11.565225+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:04:00.683556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.567512+0000","last_change":"2024-01-15T11:26:11.558952+0000","last_active":"2024-01-15T11:26:11.567512+0000","last_peered":"2024-01-15T11:26:11.567512+0000","last_clean":"2024-01-15T11:26:11.567512+0000","last_became_active":"2024-01-15T11:26:11.558659+0000","last_became_peered":"2024-01-15T11:26:11.558659+0000","last_unstale":"2024-01-15T11:26:11.567512+0000","last_undegraded":"2024-01-15T11:26:11.567512+0000","last_fullsized":"2024-01-15T11:26:11.567512+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:14:25.674690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.559457+0000","last_change":"2024-01-15T11:26:10.553845+0000","last_active":"2024-01-15T11:26:11.559457+0000","last_peered":"2024-01-15T11:26:11.559457+0000","last_clean":"2024-01-15T11:26:11.559457+0000","last_became_active":"2024-01-15T11:26:10.553580+0000","last_became_peered":"2024-01-15T11:26:10.553580+0000","last_unstale":"2024-01-15T11:26:11.559457+0000","last_undegraded":"2024-01-15T11:26:11.559457+0000","last_fullsized":"2024-01-15T11:26:11.559457+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:27:56.730362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566602+0000","last_change":"2024-01-15T11:26:11.562991+0000","last_active":"2024-01-15T11:26:11.566602+0000","last_peered":"2024-01-15T11:26:11.566602+0000","last_clean":"2024-01-15T11:26:11.566602+0000","last_became_active":"2024-01-15T11:26:11.562747+0000","last_became_peered":"2024-01-15T11:26:11.562747+0000","last_unstale":"2024-01-15T11:26:11.566602+0000","last_undegraded":"2024-01-15T11:26:11.566602+0000","last_fullsized":"2024-01-15T11:26:11.566602+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:10.490220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.568305+0000","last_change":"2024-01-15T11:26:10.564715+0000","last_active":"2024-01-15T11:26:11.568305+0000","last_peered":"2024-01-15T11:26:11.568305+0000","last_clean":"2024-01-15T11:26:11.568305+0000","last_became_active":"2024-01-15T11:26:10.564360+0000","last_became_peered":"2024-01-15T11:26:10.564360+0000","last_unstale":"2024-01-15T11:26:11.568305+0000","last_undegraded":"2024-01-15T11:26:11.568305+0000","last_fullsized":"2024-01-15T11:26:11.568305+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:15.789127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.569529+0000","last_change":"2024-01-15T11:26:11.567982+0000","last_active":"2024-01-15T11:26:11.569529+0000","last_peered":"2024-01-15T11:26:11.569529+0000","last_clean":"2024-01-15T11:26:11.569529+0000","last_became_active":"2024-01-15T11:26:11.567245+0000","last_became_peered":"2024-01-15T11:26:11.567245+0000","last_unstale":"2024-01-15T11:26:11.569529+0000","last_undegraded":"2024-01-15T11:26:11.569529+0000","last_fullsized":"2024-01-15T11:26:11.569529+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:10.490220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565680+0000","last_change":"2024-01-15T11:26:10.553909+0000","last_active":"2024-01-15T11:26:11.565680+0000","last_peered":"2024-01-15T11:26:11.565680+0000","last_clean":"2024-01-15T11:26:11.565680+0000","last_became_active":"2024-01-15T11:26:10.553692+0000","last_became_peered":"2024-01-15T11:26:10.553692+0000","last_unstale":"2024-01-15T11:26:11.565680+0000","last_undegraded":"2024-01-15T11:26:11.565680+0000","last_fullsized":"2024-01-15T11:26:11.565680+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:39.115189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.558414+0000","last_change":"2024-01-15T11:26:11.554416+0000","last_active":"2024-01-15T11:26:11.558414+0000","last_peered":"2024-01-15T11:26:11.558414+0000","last_clean":"2024-01-15T11:26:11.558414+0000","last_became_active":"2024-01-15T11:26:11.553609+0000","last_became_peered":"2024-01-15T11:26:11.553609+0000","last_unstale":"2024-01-15T11:26:11.558414+0000","last_undegraded":"2024-01-15T11:26:11.558414+0000","last_fullsized":"2024-01-15T11:26:11.558414+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:26.983127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.561951+0000","last_change":"2024-01-15T11:26:10.556678+0000","last_active":"2024-01-15T11:26:11.561951+0000","last_peered":"2024-01-15T11:26:11.561951+0000","last_clean":"2024-01-15T11:26:11.561951+0000","last_became_active":"2024-01-15T11:26:10.556409+0000","last_became_peered":"2024-01-15T11:26:10.556409+0000","last_unstale":"2024-01-15T11:26:11.561951+0000","last_undegraded":"2024-01-15T11:26:11.561951+0000","last_fullsized":"2024-01-15T11:26:11.561951+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:33:17.216079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.555396+0000","last_change":"2024-01-15T11:26:11.553706+0000","last_active":"2024-01-15T11:26:11.555396+0000","last_peered":"2024-01-15T11:26:11.555396+0000","last_clean":"2024-01-15T11:26:11.555396+0000","last_became_active":"2024-01-15T11:26:11.553339+0000","last_became_peered":"2024-01-15T11:26:11.553339+0000","last_unstale":"2024-01-15T11:26:11.555396+0000","last_undegraded":"2024-01-15T11:26:11.555396+0000","last_fullsized":"2024-01-15T11:26:11.555396+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:45:31.756657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.562018+0000","last_change":"2024-01-15T11:26:10.554498+0000","last_active":"2024-01-15T11:26:11.562018+0000","last_peered":"2024-01-15T11:26:11.562018+0000","last_clean":"2024-01-15T11:26:11.562018+0000","last_became_active":"2024-01-15T11:26:10.554138+0000","last_became_peered":"2024-01-15T11:26:10.554138+0000","last_unstale":"2024-01-15T11:26:11.562018+0000","last_undegraded":"2024-01-15T11:26:11.562018+0000","last_fullsized":"2024-01-15T11:26:11.562018+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:38.540135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566322+0000","last_change":"2024-01-15T11:26:11.560035+0000","last_active":"2024-01-15T11:26:11.566322+0000","last_peered":"2024-01-15T11:26:11.566322+0000","last_clean":"2024-01-15T11:26:11.566322+0000","last_became_active":"2024-01-15T11:26:11.559762+0000","last_became_peered":"2024-01-15T11:26:11.559762+0000","last_unstale":"2024-01-15T11:26:11.566322+0000","last_undegraded":"2024-01-15T11:26:11.566322+0000","last_fullsized":"2024-01-15T11:26:11.566322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:40.120702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.541452+0000","last_change":"2024-01-15T11:26:10.545920+0000","last_active":"2024-01-15T11:26:11.541452+0000","last_peered":"2024-01-15T11:26:11.541452+0000","last_clean":"2024-01-15T11:26:11.541452+0000","last_became_active":"2024-01-15T11:26:10.545643+0000","last_became_peered":"2024-01-15T11:26:10.545643+0000","last_unstale":"2024-01-15T11:26:11.541452+0000","last_undegraded":"2024-01-15T11:26:11.541452+0000","last_fullsized":"2024-01-15T11:26:11.541452+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T12:55:09.992378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565406+0000","last_change":"2024-01-15T11:26:11.557757+0000","last_active":"2024-01-15T11:26:11.565406+0000","last_peered":"2024-01-15T11:26:11.565406+0000","last_clean":"2024-01-15T11:26:11.565406+0000","last_became_active":"2024-01-15T11:26:11.557499+0000","last_became_peered":"2024-01-15T11:26:11.557499+0000","last_unstale":"2024-01-15T11:26:11.565406+0000","last_undegraded":"2024-01-15T11:26:11.565406+0000","last_fullsized":"2024-01-15T11:26:11.565406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:18.187321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.547816+0000","last_change":"2024-01-15T11:26:10.556424+0000","last_active":"2024-01-15T11:26:11.547816+0000","last_peered":"2024-01-15T11:26:11.547816+0000","last_clean":"2024-01-15T11:26:11.547816+0000","last_became_active":"2024-01-15T11:26:10.556144+0000","last_became_peered":"2024-01-15T11:26:10.556144+0000","last_unstale":"2024-01-15T11:26:11.547816+0000","last_undegraded":"2024-01-15T11:26:11.547816+0000","last_fullsized":"2024-01-15T11:26:11.547816+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:32:42.884039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.555259+0000","last_change":"2024-01-15T11:26:11.552356+0000","last_active":"2024-01-15T11:26:11.555259+0000","last_peered":"2024-01-15T11:26:11.555259+0000","last_clean":"2024-01-15T11:26:11.555259+0000","last_became_active":"2024-01-15T11:26:11.552051+0000","last_became_peered":"2024-01-15T11:26:11.552051+0000","last_unstale":"2024-01-15T11:26:11.555259+0000","last_undegraded":"2024-01-15T11:26:11.555259+0000","last_fullsized":"2024-01-15T11:26:11.555259+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:12:09.208938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.544001+0000","last_change":"2024-01-15T11:26:10.546384+0000","last_active":"2024-01-15T11:26:11.544001+0000","last_peered":"2024-01-15T11:26:11.544001+0000","last_clean":"2024-01-15T11:26:11.544001+0000","last_became_active":"2024-01-15T11:26:10.545705+0000","last_became_peered":"2024-01-15T11:26:10.545705+0000","last_unstale":"2024-01-15T11:26:11.544001+0000","last_undegraded":"2024-01-15T11:26:11.544001+0000","last_fullsized":"2024-01-15T11:26:11.544001+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:53.089530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.571064+0000","last_change":"2024-01-15T11:26:11.567130+0000","last_active":"2024-01-15T11:26:11.571064+0000","last_peered":"2024-01-15T11:26:11.571064+0000","last_clean":"2024-01-15T11:26:11.571064+0000","last_became_active":"2024-01-15T11:26:11.563343+0000","last_became_peered":"2024-01-15T11:26:11.563343+0000","last_unstale":"2024-01-15T11:26:11.571064+0000","last_undegraded":"2024-01-15T11:26:11.571064+0000","last_fullsized":"2024-01-15T11:26:11.571064+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:02:39.352286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.550077+0000","last_change":"2024-01-15T11:26:10.552843+0000","last_active":"2024-01-15T11:26:11.550077+0000","last_peered":"2024-01-15T11:26:11.550077+0000","last_clean":"2024-01-15T11:26:11.550077+0000","last_became_active":"2024-01-15T11:26:10.552556+0000","last_became_peered":"2024-01-15T11:26:10.552556+0000","last_unstale":"2024-01-15T11:26:11.550077+0000","last_undegraded":"2024-01-15T11:26:11.550077+0000","last_fullsized":"2024-01-15T11:26:11.550077+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:38.540135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560573+0000","last_change":"2024-01-15T11:26:11.555669+0000","last_active":"2024-01-15T11:26:11.560573+0000","last_peered":"2024-01-15T11:26:11.560573+0000","last_clean":"2024-01-15T11:26:11.560573+0000","last_became_active":"2024-01-15T11:26:11.555423+0000","last_became_peered":"2024-01-15T11:26:11.555423+0000","last_unstale":"2024-01-15T11:26:11.560573+0000","last_undegraded":"2024-01-15T11:26:11.560573+0000","last_fullsized":"2024-01-15T11:26:11.560573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:26:19.933700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.555760+0000","last_change":"2024-01-15T11:26:10.550026+0000","last_active":"2024-01-15T11:26:11.555760+0000","last_peered":"2024-01-15T11:26:11.555760+0000","last_clean":"2024-01-15T11:26:11.555760+0000","last_became_active":"2024-01-15T11:26:10.549672+0000","last_became_peered":"2024-01-15T11:26:10.549672+0000","last_unstale":"2024-01-15T11:26:11.555760+0000","last_undegraded":"2024-01-15T11:26:11.555760+0000","last_fullsized":"2024-01-15T11:26:11.555760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:54:27.570386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566817+0000","last_change":"2024-01-15T11:26:11.564314+0000","last_active":"2024-01-15T11:26:11.566817+0000","last_peered":"2024-01-15T11:26:11.566817+0000","last_clean":"2024-01-15T11:26:11.566817+0000","last_became_active":"2024-01-15T11:26:11.563976+0000","last_became_peered":"2024-01-15T11:26:11.563976+0000","last_unstale":"2024-01-15T11:26:11.566817+0000","last_undegraded":"2024-01-15T11:26:11.566817+0000","last_fullsized":"2024-01-15T11:26:11.566817+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:51:14.891621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.572289+0000","last_change":"2024-01-15T11:26:10.555654+0000","last_active":"2024-01-15T11:26:11.572289+0000","last_peered":"2024-01-15T11:26:11.572289+0000","last_clean":"2024-01-15T11:26:11.572289+0000","last_became_active":"2024-01-15T11:26:10.555265+0000","last_became_peered":"2024-01-15T11:26:10.555265+0000","last_unstale":"2024-01-15T11:26:11.572289+0000","last_undegraded":"2024-01-15T11:26:11.572289+0000","last_fullsized":"2024-01-15T11:26:11.572289+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:11.725891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565994+0000","last_change":"2024-01-15T11:26:11.558260+0000","last_active":"2024-01-15T11:26:11.565994+0000","last_peered":"2024-01-15T11:26:11.565994+0000","last_clean":"2024-01-15T11:26:11.565994+0000","last_became_active":"2024-01-15T11:26:11.558016+0000","last_became_peered":"2024-01-15T11:26:11.558016+0000","last_unstale":"2024-01-15T11:26:11.565994+0000","last_undegraded":"2024-01-15T11:26:11.565994+0000","last_fullsized":"2024-01-15T11:26:11.565994+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:34:27.955198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565586+0000","last_change":"2024-01-15T11:26:10.554161+0000","last_active":"2024-01-15T11:26:11.565586+0000","last_peered":"2024-01-15T11:26:11.565586+0000","last_clean":"2024-01-15T11:26:11.565586+0000","last_became_active":"2024-01-15T11:26:10.553835+0000","last_became_peered":"2024-01-15T11:26:10.553835+0000","last_unstale":"2024-01-15T11:26:11.565586+0000","last_undegraded":"2024-01-15T11:26:11.565586+0000","last_fullsized":"2024-01-15T11:26:11.565586+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T12:55:09.992378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560535+0000","last_change":"2024-01-15T11:26:11.553748+0000","last_active":"2024-01-15T11:26:11.560535+0000","last_peered":"2024-01-15T11:26:11.560535+0000","last_clean":"2024-01-15T11:26:11.560535+0000","last_became_active":"2024-01-15T11:26:11.553361+0000","last_became_peered":"2024-01-15T11:26:11.553361+0000","last_unstale":"2024-01-15T11:26:11.560535+0000","last_undegraded":"2024-01-15T11:26:11.560535+0000","last_fullsized":"2024-01-15T11:26:11.560535+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:12:09.208938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.554659+0000","last_change":"2024-01-15T11:26:10.556577+0000","last_active":"2024-01-15T11:26:11.554659+0000","last_peered":"2024-01-15T11:26:11.554659+0000","last_clean":"2024-01-15T11:26:11.554659+0000","last_became_active":"2024-01-15T11:26:10.556263+0000","last_became_peered":"2024-01-15T11:26:10.556263+0000","last_unstale":"2024-01-15T11:26:11.554659+0000","last_undegraded":"2024-01-15T11:26:11.554659+0000","last_fullsized":"2024-01-15T11:26:11.554659+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:33:17.216079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.559606+0000","last_change":"2024-01-15T11:26:11.554161+0000","last_active":"2024-01-15T11:26:11.559606+0000","last_peered":"2024-01-15T11:26:11.559606+0000","last_clean":"2024-01-15T11:26:11.559606+0000","last_became_active":"2024-01-15T11:26:11.553854+0000","last_became_peered":"2024-01-15T11:26:11.553854+0000","last_unstale":"2024-01-15T11:26:11.559606+0000","last_undegraded":"2024-01-15T11:26:11.559606+0000","last_fullsized":"2024-01-15T11:26:11.559606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:40:49.107290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.541347+0000","last_change":"2024-01-15T11:26:10.544207+0000","last_active":"2024-01-15T11:26:11.541347+0000","last_peered":"2024-01-15T11:26:11.541347+0000","last_clean":"2024-01-15T11:26:11.541347+0000","last_became_active":"2024-01-15T11:26:10.543937+0000","last_became_peered":"2024-01-15T11:26:10.543937+0000","last_unstale":"2024-01-15T11:26:11.541347+0000","last_undegraded":"2024-01-15T11:26:11.541347+0000","last_fullsized":"2024-01-15T11:26:11.541347+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:51.789623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.567775+0000","last_change":"2024-01-15T11:26:11.560138+0000","last_active":"2024-01-15T11:26:11.567775+0000","last_peered":"2024-01-15T11:26:11.567775+0000","last_clean":"2024-01-15T11:26:11.567775+0000","last_became_active":"2024-01-15T11:26:11.559673+0000","last_became_peered":"2024-01-15T11:26:11.559673+0000","last_unstale":"2024-01-15T11:26:11.567775+0000","last_undegraded":"2024-01-15T11:26:11.567775+0000","last_fullsized":"2024-01-15T11:26:11.567775+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:27:57.735875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.569532+0000","last_change":"2024-01-15T11:26:10.556115+0000","last_active":"2024-01-15T11:26:11.569532+0000","last_peered":"2024-01-15T11:26:11.569532+0000","last_clean":"2024-01-15T11:26:11.569532+0000","last_became_active":"2024-01-15T11:26:10.555211+0000","last_became_peered":"2024-01-15T11:26:10.555211+0000","last_unstale":"2024-01-15T11:26:11.569532+0000","last_undegraded":"2024-01-15T11:26:11.569532+0000","last_fullsized":"2024-01-15T11:26:11.569532+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:45:30.751144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.561580+0000","last_change":"2024-01-15T11:26:11.555885+0000","last_active":"2024-01-15T11:26:11.561580+0000","last_peered":"2024-01-15T11:26:11.561580+0000","last_clean":"2024-01-15T11:26:11.561580+0000","last_became_active":"2024-01-15T11:26:11.555621+0000","last_became_peered":"2024-01-15T11:26:11.555621+0000","last_unstale":"2024-01-15T11:26:11.561580+0000","last_undegraded":"2024-01-15T11:26:11.561580+0000","last_fullsized":"2024-01-15T11:26:11.561580+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:18.187321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.571242+0000","last_change":"2024-01-15T11:26:10.565356+0000","last_active":"2024-01-15T11:26:11.571242+0000","last_peered":"2024-01-15T11:26:11.571242+0000","last_clean":"2024-01-15T11:26:11.571242+0000","last_became_active":"2024-01-15T11:26:10.564922+0000","last_became_peered":"2024-01-15T11:26:10.564922+0000","last_unstale":"2024-01-15T11:26:11.571242+0000","last_undegraded":"2024-01-15T11:26:11.571242+0000","last_fullsized":"2024-01-15T11:26:11.571242+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:45:30.751144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.563546+0000","last_change":"2024-01-15T11:26:11.558248+0000","last_active":"2024-01-15T11:26:11.563546+0000","last_peered":"2024-01-15T11:26:11.563546+0000","last_clean":"2024-01-15T11:26:11.563546+0000","last_became_active":"2024-01-15T11:26:11.556305+0000","last_became_peered":"2024-01-15T11:26:11.556305+0000","last_unstale":"2024-01-15T11:26:11.563546+0000","last_undegraded":"2024-01-15T11:26:11.563546+0000","last_fullsized":"2024-01-15T11:26:11.563546+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:59:32.790611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560348+0000","last_change":"2024-01-15T11:26:10.555836+0000","last_active":"2024-01-15T11:26:11.560348+0000","last_peered":"2024-01-15T11:26:11.560348+0000","last_clean":"2024-01-15T11:26:11.560348+0000","last_became_active":"2024-01-15T11:26:10.555554+0000","last_became_peered":"2024-01-15T11:26:10.555554+0000","last_unstale":"2024-01-15T11:26:11.560348+0000","last_undegraded":"2024-01-15T11:26:11.560348+0000","last_fullsized":"2024-01-15T11:26:11.560348+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:23:16.009131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566043+0000","last_change":"2024-01-15T11:26:11.557802+0000","last_active":"2024-01-15T11:26:11.566043+0000","last_peered":"2024-01-15T11:26:11.566043+0000","last_clean":"2024-01-15T11:26:11.566043+0000","last_became_active":"2024-01-15T11:26:11.557536+0000","last_became_peered":"2024-01-15T11:26:11.557536+0000","last_unstale":"2024-01-15T11:26:11.566043+0000","last_undegraded":"2024-01-15T11:26:11.566043+0000","last_fullsized":"2024-01-15T11:26:11.566043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:32:43.889552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.563392+0000","last_change":"2024-01-15T11:26:10.555888+0000","last_active":"2024-01-15T11:26:11.563392+0000","last_peered":"2024-01-15T11:26:11.563392+0000","last_clean":"2024-01-15T11:26:11.563392+0000","last_became_active":"2024-01-15T11:26:10.555535+0000","last_became_peered":"2024-01-15T11:26:10.555535+0000","last_unstale":"2024-01-15T11:26:11.563392+0000","last_undegraded":"2024-01-15T11:26:11.563392+0000","last_fullsized":"2024-01-15T11:26:11.563392+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:04:00.683556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564926+0000","last_change":"2024-01-15T11:26:11.556887+0000","last_active":"2024-01-15T11:26:11.564926+0000","last_peered":"2024-01-15T11:26:11.564926+0000","last_clean":"2024-01-15T11:26:11.564926+0000","last_became_active":"2024-01-15T11:26:11.556494+0000","last_became_peered":"2024-01-15T11:26:11.556494+0000","last_unstale":"2024-01-15T11:26:11.564926+0000","last_undegraded":"2024-01-15T11:26:11.564926+0000","last_fullsized":"2024-01-15T11:26:11.564926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:29:24.215640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.552199+0000","last_change":"2024-01-15T11:26:10.554108+0000","last_active":"2024-01-15T11:26:11.552199+0000","last_peered":"2024-01-15T11:26:11.552199+0000","last_clean":"2024-01-15T11:26:11.552199+0000","last_became_active":"2024-01-15T11:26:10.553801+0000","last_became_peered":"2024-01-15T11:26:10.553801+0000","last_unstale":"2024-01-15T11:26:11.552199+0000","last_undegraded":"2024-01-15T11:26:11.552199+0000","last_fullsized":"2024-01-15T11:26:11.552199+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:26:18.928187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564871+0000","last_change":"2024-01-15T11:26:11.557103+0000","last_active":"2024-01-15T11:26:11.564871+0000","last_peered":"2024-01-15T11:26:11.564871+0000","last_clean":"2024-01-15T11:26:11.564871+0000","last_became_active":"2024-01-15T11:26:11.556759+0000","last_became_peered":"2024-01-15T11:26:11.556759+0000","last_unstale":"2024-01-15T11:26:11.564871+0000","last_undegraded":"2024-01-15T11:26:11.564871+0000","last_fullsized":"2024-01-15T11:26:11.564871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:48:00.776358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.563430+0000","last_change":"2024-01-15T11:26:10.553997+0000","last_active":"2024-01-15T11:26:11.563430+0000","last_peered":"2024-01-15T11:26:11.563430+0000","last_clean":"2024-01-15T11:26:11.563430+0000","last_became_active":"2024-01-15T11:26:10.553687+0000","last_became_peered":"2024-01-15T11:26:10.553687+0000","last_unstale":"2024-01-15T11:26:11.563430+0000","last_undegraded":"2024-01-15T11:26:11.563430+0000","last_fullsized":"2024-01-15T11:26:11.563430+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:23:16.009131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565169+0000","last_change":"2024-01-15T11:26:11.562925+0000","last_active":"2024-01-15T11:26:11.565169+0000","last_peered":"2024-01-15T11:26:11.565169+0000","last_clean":"2024-01-15T11:26:11.565169+0000","last_became_active":"2024-01-15T11:26:11.562651+0000","last_became_peered":"2024-01-15T11:26:11.562651+0000","last_unstale":"2024-01-15T11:26:11.565169+0000","last_undegraded":"2024-01-15T11:26:11.565169+0000","last_fullsized":"2024-01-15T11:26:11.565169+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:19:22.144552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.551014+0000","last_change":"2024-01-15T11:26:10.553741+0000","last_active":"2024-01-15T11:26:11.551014+0000","last_peered":"2024-01-15T11:26:11.551014+0000","last_clean":"2024-01-15T11:26:11.551014+0000","last_became_active":"2024-01-15T11:26:10.553432+0000","last_became_peered":"2024-01-15T11:26:10.553432+0000","last_unstale":"2024-01-15T11:26:11.551014+0000","last_undegraded":"2024-01-15T11:26:11.551014+0000","last_fullsized":"2024-01-15T11:26:11.551014+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:17.181809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565262+0000","last_change":"2024-01-15T11:26:11.558534+0000","last_active":"2024-01-15T11:26:11.565262+0000","last_peered":"2024-01-15T11:26:11.565262+0000","last_clean":"2024-01-15T11:26:11.565262+0000","last_became_active":"2024-01-15T11:26:11.556712+0000","last_became_peered":"2024-01-15T11:26:11.556712+0000","last_unstale":"2024-01-15T11:26:11.565262+0000","last_undegraded":"2024-01-15T11:26:11.565262+0000","last_fullsized":"2024-01-15T11:26:11.565262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:59:32.655545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.546323+0000","last_change":"2024-01-15T11:26:10.556697+0000","last_active":"2024-01-15T11:26:11.546323+0000","last_peered":"2024-01-15T11:26:11.546323+0000","last_clean":"2024-01-15T11:26:11.546323+0000","last_became_active":"2024-01-15T11:26:10.555974+0000","last_became_peered":"2024-01-15T11:26:10.555974+0000","last_unstale":"2024-01-15T11:26:11.546323+0000","last_undegraded":"2024-01-15T11:26:11.546323+0000","last_fullsized":"2024-01-15T11:26:11.546323+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:14:24.669178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564316+0000","last_change":"2024-01-15T11:26:11.561846+0000","last_active":"2024-01-15T11:26:11.564316+0000","last_peered":"2024-01-15T11:26:11.564316+0000","last_clean":"2024-01-15T11:26:11.564316+0000","last_became_active":"2024-01-15T11:26:11.561245+0000","last_became_peered":"2024-01-15T11:26:11.561245+0000","last_unstale":"2024-01-15T11:26:11.564316+0000","last_undegraded":"2024-01-15T11:26:11.564316+0000","last_fullsized":"2024-01-15T11:26:11.564316+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:51:14.891621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.542692+0000","last_change":"2024-01-15T11:26:10.549311+0000","last_active":"2024-01-15T11:26:11.542692+0000","last_peered":"2024-01-15T11:26:11.542692+0000","last_clean":"2024-01-15T11:26:11.542692+0000","last_became_active":"2024-01-15T11:26:10.549026+0000","last_became_peered":"2024-01-15T11:26:10.549026+0000","last_unstale":"2024-01-15T11:26:11.542692+0000","last_undegraded":"2024-01-15T11:26:11.542692+0000","last_fullsized":"2024-01-15T11:26:11.542692+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:15.789127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564941+0000","last_change":"2024-01-15T11:26:11.557587+0000","last_active":"2024-01-15T11:26:11.564941+0000","last_peered":"2024-01-15T11:26:11.564941+0000","last_clean":"2024-01-15T11:26:11.564941+0000","last_became_active":"2024-01-15T11:26:11.556932+0000","last_became_peered":"2024-01-15T11:26:11.556932+0000","last_unstale":"2024-01-15T11:26:11.564941+0000","last_undegraded":"2024-01-15T11:26:11.564941+0000","last_fullsized":"2024-01-15T11:26:11.564941+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:23:24.437335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.569917+0000","last_change":"2024-01-15T11:26:10.565465+0000","last_active":"2024-01-15T11:26:11.569917+0000","last_peered":"2024-01-15T11:26:11.569917+0000","last_clean":"2024-01-15T11:26:11.569917+0000","last_became_active":"2024-01-15T11:26:10.565175+0000","last_became_peered":"2024-01-15T11:26:10.565175+0000","last_unstale":"2024-01-15T11:26:11.569917+0000","last_undegraded":"2024-01-15T11:26:11.569917+0000","last_fullsized":"2024-01-15T11:26:11.569917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:51.789623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566522+0000","last_change":"2024-01-15T11:26:11.564289+0000","last_active":"2024-01-15T11:26:11.566522+0000","last_peered":"2024-01-15T11:26:11.566522+0000","last_clean":"2024-01-15T11:26:11.566522+0000","last_became_active":"2024-01-15T11:26:11.564008+0000","last_became_peered":"2024-01-15T11:26:11.564008+0000","last_unstale":"2024-01-15T11:26:11.566522+0000","last_undegraded":"2024-01-15T11:26:11.566522+0000","last_fullsized":"2024-01-15T11:26:11.566522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:40:49.107290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.550217+0000","last_change":"2024-01-15T11:26:10.554135+0000","last_active":"2024-01-15T11:26:11.550217+0000","last_peered":"2024-01-15T11:26:11.550217+0000","last_clean":"2024-01-15T11:26:11.550217+0000","last_became_active":"2024-01-15T11:26:10.553831+0000","last_became_peered":"2024-01-15T11:26:10.553831+0000","last_unstale":"2024-01-15T11:26:11.550217+0000","last_undegraded":"2024-01-15T11:26:11.550217+0000","last_fullsized":"2024-01-15T11:26:11.550217+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:04:00.683556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566826+0000","last_change":"2024-01-15T11:26:11.565505+0000","last_active":"2024-01-15T11:26:11.566826+0000","last_peered":"2024-01-15T11:26:11.566826+0000","last_clean":"2024-01-15T11:26:11.566826+0000","last_became_active":"2024-01-15T11:26:11.565196+0000","last_became_peered":"2024-01-15T11:26:11.565196+0000","last_unstale":"2024-01-15T11:26:11.566826+0000","last_undegraded":"2024-01-15T11:26:11.566826+0000","last_fullsized":"2024-01-15T11:26:11.566826+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:34:27.955198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.555696+0000","last_change":"2024-01-15T11:26:10.555925+0000","last_active":"2024-01-15T11:26:11.555696+0000","last_peered":"2024-01-15T11:26:11.555696+0000","last_clean":"2024-01-15T11:26:11.555696+0000","last_became_active":"2024-01-15T11:26:10.555448+0000","last_became_peered":"2024-01-15T11:26:10.555448+0000","last_unstale":"2024-01-15T11:26:11.555696+0000","last_undegraded":"2024-01-15T11:26:11.555696+0000","last_fullsized":"2024-01-15T11:26:11.555696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:26:18.928187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.567620+0000","last_change":"2024-01-15T11:26:11.559204+0000","last_active":"2024-01-15T11:26:11.567620+0000","last_peered":"2024-01-15T11:26:11.567620+0000","last_clean":"2024-01-15T11:26:11.567620+0000","last_became_active":"2024-01-15T11:26:11.558824+0000","last_became_peered":"2024-01-15T11:26:11.558824+0000","last_unstale":"2024-01-15T11:26:11.567620+0000","last_undegraded":"2024-01-15T11:26:11.567620+0000","last_fullsized":"2024-01-15T11:26:11.567620+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:35:16.780032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.550825+0000","last_change":"2024-01-15T11:26:10.551338+0000","last_active":"2024-01-15T11:26:11.550825+0000","last_peered":"2024-01-15T11:26:11.550825+0000","last_clean":"2024-01-15T11:26:11.550825+0000","last_became_active":"2024-01-15T11:26:10.551039+0000","last_became_peered":"2024-01-15T11:26:10.551039+0000","last_unstale":"2024-01-15T11:26:11.550825+0000","last_undegraded":"2024-01-15T11:26:11.550825+0000","last_fullsized":"2024-01-15T11:26:11.550825+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:09.484707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.561094+0000","last_change":"2024-01-15T11:26:11.556659+0000","last_active":"2024-01-15T11:26:11.561094+0000","last_peered":"2024-01-15T11:26:11.561094+0000","last_clean":"2024-01-15T11:26:11.561094+0000","last_became_active":"2024-01-15T11:26:11.556404+0000","last_became_peered":"2024-01-15T11:26:11.556404+0000","last_unstale":"2024-01-15T11:26:11.561094+0000","last_undegraded":"2024-01-15T11:26:11.561094+0000","last_fullsized":"2024-01-15T11:26:11.561094+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:32:43.889552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.554738+0000","last_change":"2024-01-15T11:26:10.559269+0000","last_active":"2024-01-15T11:26:11.554738+0000","last_peered":"2024-01-15T11:26:11.554738+0000","last_clean":"2024-01-15T11:26:11.554738+0000","last_became_active":"2024-01-15T11:26:10.558981+0000","last_became_peered":"2024-01-15T11:26:10.558981+0000","last_unstale":"2024-01-15T11:26:11.554738+0000","last_undegraded":"2024-01-15T11:26:11.554738+0000","last_fullsized":"2024-01-15T11:26:11.554738+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T12:55:09.992378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560672+0000","last_change":"2024-01-15T11:26:11.558476+0000","last_active":"2024-01-15T11:26:11.560672+0000","last_peered":"2024-01-15T11:26:11.560672+0000","last_clean":"2024-01-15T11:26:11.560672+0000","last_became_active":"2024-01-15T11:26:11.558222+0000","last_became_peered":"2024-01-15T11:26:11.558222+0000","last_unstale":"2024-01-15T11:26:11.560672+0000","last_undegraded":"2024-01-15T11:26:11.560672+0000","last_fullsized":"2024-01-15T11:26:11.560672+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:33:18.221592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.546480+0000","last_change":"2024-01-15T11:26:10.556277+0000","last_active":"2024-01-15T11:26:11.546480+0000","last_peered":"2024-01-15T11:26:11.546480+0000","last_clean":"2024-01-15T11:26:11.546480+0000","last_became_active":"2024-01-15T11:26:10.555654+0000","last_became_peered":"2024-01-15T11:26:10.555654+0000","last_unstale":"2024-01-15T11:26:11.546480+0000","last_undegraded":"2024-01-15T11:26:11.546480+0000","last_fullsized":"2024-01-15T11:26:11.546480+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:34:26.949685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566511+0000","last_change":"2024-01-15T11:26:11.562207+0000","last_active":"2024-01-15T11:26:11.566511+0000","last_peered":"2024-01-15T11:26:11.566511+0000","last_clean":"2024-01-15T11:26:11.566511+0000","last_became_active":"2024-01-15T11:26:11.561896+0000","last_became_peered":"2024-01-15T11:26:11.561896+0000","last_unstale":"2024-01-15T11:26:11.566511+0000","last_undegraded":"2024-01-15T11:26:11.566511+0000","last_fullsized":"2024-01-15T11:26:11.566511+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:47:02.390456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.541680+0000","last_change":"2024-01-15T11:26:10.545067+0000","last_active":"2024-01-15T11:26:11.541680+0000","last_peered":"2024-01-15T11:26:11.541680+0000","last_clean":"2024-01-15T11:26:11.541680+0000","last_became_active":"2024-01-15T11:26:10.544775+0000","last_became_peered":"2024-01-15T11:26:10.544775+0000","last_unstale":"2024-01-15T11:26:11.541680+0000","last_undegraded":"2024-01-15T11:26:11.541680+0000","last_fullsized":"2024-01-15T11:26:11.541680+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:09:10.704656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564176+0000","last_change":"2024-01-15T11:26:11.560649+0000","last_active":"2024-01-15T11:26:11.564176+0000","last_peered":"2024-01-15T11:26:11.564176+0000","last_clean":"2024-01-15T11:26:11.564176+0000","last_became_active":"2024-01-15T11:26:11.560057+0000","last_became_peered":"2024-01-15T11:26:11.560057+0000","last_unstale":"2024-01-15T11:26:11.564176+0000","last_undegraded":"2024-01-15T11:26:11.564176+0000","last_fullsized":"2024-01-15T11:26:11.564176+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:59:32.790611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.551096+0000","last_change":"2024-01-15T11:26:10.557573+0000","last_active":"2024-01-15T11:26:11.551096+0000","last_peered":"2024-01-15T11:26:11.551096+0000","last_clean":"2024-01-15T11:26:11.551096+0000","last_became_active":"2024-01-15T11:26:10.557279+0000","last_became_peered":"2024-01-15T11:26:10.557279+0000","last_unstale":"2024-01-15T11:26:11.551096+0000","last_undegraded":"2024-01-15T11:26:11.551096+0000","last_fullsized":"2024-01-15T11:26:11.551096+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:34:26.949685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-15T11:26:10.546618+0000","last_change":"2024-01-15T11:25:03.506154+0000","last_active":"2024-01-15T11:26:10.546618+0000","last_peered":"2024-01-15T11:26:10.546618+0000","last_clean":"2024-01-15T11:26:10.546618+0000","last_became_active":"2024-01-15T11:25:03.504996+0000","last_became_peered":"2024-01-15T11:25:03.504996+0000","last_unstale":"2024-01-15T11:26:10.546618+0000","last_undegraded":"2024-01-15T11:26:10.546618+0000","last_fullsized":"2024-01-15T11:26:10.546618+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:02.494136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:02.494136+0000","last_clean_scrub_stamp":"2024-01-15T11:25:02.494136+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:22:33.043449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":723877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":3.371,"5min":3.371,"15min":3.371},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":3.4420000000000002,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":0.53200000000000003}]},{"osd":1,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.0609999999999999,"5min":6.0609999999999999,"15min":6.0609999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":10.359,"5min":10.359,"15min":10.359},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.5190000000000001,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.56599999999999995}]},{"osd":3,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":10.523,"5min":10.523,"15min":10.523},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.133,"5min":1.133,"15min":1.133},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":10.497999999999999,"5min":10.497999999999999,"15min":10.497999999999999},"last":0.68999999999999995}]},{"osd":4,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.42299999999999999}]},{"osd":5,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.371},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.40100000000000002}]},{"osd":6,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.495}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":368640,"data_stored":140900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.59699999999999998}]},{"osd":1,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.64900000000000002}]},{"osd":2,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.51200000000000001}]},{"osd":3,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.624},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66600000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":141918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":4.0620000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.45400000000000001}]},{"osd":2,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.46,"5min":4.46,"15min":4.46},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.9470000000000001,"5min":3.9470000000000001,"15min":3.9470000000000001},"last":0.55100000000000005}]},{"osd":3,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.6799999999999997,"5min":4.6799999999999997,"15min":4.6799999999999997},"last":4.1299999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":4.7190000000000003}]},{"osd":4,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":4.3620000000000001,"5min":4.3620000000000001,"15min":4.3620000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.9940000000000002,"5min":3.9940000000000002,"15min":3.9940000000000002},"last":0.497}]},{"osd":5,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.3979999999999997,"5min":4.3979999999999997,"15min":4.3979999999999997},"last":0.69999999999999996}]},{"osd":6,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":4.4290000000000003,"5min":4.4290000000000003,"15min":4.4290000000000003},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":4.0259999999999998,"5min":4.0259999999999998,"15min":4.0259999999999998},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":1.012}]}]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":723877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"last":0.216},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.372}]},{"osd":2,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"last":0.252},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"last":0.32000000000000001}]},{"osd":4,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.67800000000000005}]},{"osd":6,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.52200000000000002}]},{"osd":7,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.57499999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.109,"5min":5.109,"15min":5.109},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.2380000000000004,"5min":5.2380000000000004,"15min":5.2380000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":3.5910000000000002,"5min":3.5910000000000002,"15min":3.5910000000000002},"last":0.41099999999999998}]},{"osd":3,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":5.2649999999999997,"5min":5.2649999999999997,"15min":5.2649999999999997},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":3.6720000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":0.63}]},{"osd":5,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":5.2110000000000003,"5min":5.2110000000000003,"15min":5.2110000000000003},"last":0.58199999999999996}]},{"osd":6,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.7000000000000002,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":5.1500000000000004,"5min":5.1500000000000004,"15min":5.1500000000000004},"last":0.622}]},{"osd":7,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.6349999999999998,"5min":3.6349999999999998,"15min":3.6349999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.61199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.33800000000000002}]},{"osd":1,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.307}]},{"osd":2,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.36199999999999999}]},{"osd":4,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.57799999999999996}]},{"osd":5,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.54400000000000004}]},{"osd":6,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.51100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574864,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":368640,"data_stored":140388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.70199999999999996}]},{"osd":1,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.58199999999999996}]},{"osd":2,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.621},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.68500000000000005}]},{"osd":3,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.67000000000000004}]},{"osd":5,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44600000000000001}]},{"osd":6,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.504},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.434}]},{"osd":7,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.53200000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.8620000000000001,"5min":2.8620000000000001,"15min":2.8620000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.1139999999999999,"5min":5.1139999999999999,"15min":5.1139999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":0.47099999999999997}]},{"osd":2,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":7.4589999999999996,"5min":7.4589999999999996,"15min":7.4589999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":5.0350000000000001,"5min":5.0350000000000001,"15min":5.0350000000000001},"last":0.436}]},{"osd":3,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0569999999999999,"5min":3.0569999999999999,"15min":3.0569999999999999},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.501},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.36299999999999999}]},{"osd":7,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T11:26:15.662 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T11:26:15.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.819+0000 7f0c368ac700 1 Processor -- start 2024-01-15T11:26:15.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.819+0000 7f0c368ac700 1 -- start start 2024-01-15T11:26:15.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.820+0000 7f0c368ac700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0c30073f90 0x7f0c3006dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.820+0000 7f0c368ac700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c30074370 0x7f0c3015f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.820+0000 7f0c368ac700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c300cf060 con 0x7f0c3015fe20 2024-01-15T11:26:15.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.820+0000 7f0c368ac700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c301608f0 con 0x7f0c30073f90 2024-01-15T11:26:15.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.820+0000 7f0c368ac700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c300cf6e0 con 0x7f0c30074370 2024-01-15T11:26:15.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.820+0000 7f0c34e49700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f0c3015fe20 0x7f0c300cfb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56468/0 (socket says 172.21.15.113:56468) 2024-01-15T11:26:15.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.820+0000 7f0c34e49700 1 -- 172.21.15.113:0/1690575181 learned_addr learned my addr 172.21.15.113:0/1690575181 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:15.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.820+0000 7f0c2ffff700 1 --2- 172.21.15.113:0/1690575181 >> v2:172.21.15.141:3300/0 conn(0x7f0c30073f90 0x7f0c3006dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2f7fe700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c30074370 0x7f0c3015f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.820+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284045658 0 0) 0x7f0c300cf060 con 0x7f0c3015fe20 2024-01-15T11:26:15.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c1c0035b0 con 0x7f0c3015fe20 2024-01-15T11:26:15.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2770914310 0 0) 0x7f0c1c0035b0 con 0x7f0c3015fe20 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c30074370 msgr2=0x7f0c3015f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c30074370 0x7f0c3015f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 >> v2:172.21.15.141:3300/0 conn(0x7f0c30073f90 msgr2=0x7f0c3006dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 --2- 172.21.15.113:0/1690575181 >> v2:172.21.15.141:3300/0 conn(0x7f0c30073f90 0x7f0c3006dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c30160330 con 0x7f0c3015fe20 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f0c280019f0 con 0x7f0c3015fe20 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 >> v1:172.21.15.113:6789/0 conn(0x7f0c3015fe20 legacy=0x7f0c300cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c1c003a00 0x7f0c1c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c30074370 0x7f0c1c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c1c004d90 0x7f0c1c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c301608f0 con 0x7f0c1c003a00 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c300cf6e0 con 0x7f0c1c004d90 2024-01-15T11:26:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c00ba10 con 0x7f0c30074370 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2ffff700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c1c003a00 0x7f0c1c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2f7fe700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c30074370 0x7f0c1c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c34e49700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c1c004d90 0x7f0c1c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2ffff700 1 -- 172.21.15.113:0/1690575181 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c30074370 msgr2=0x7f0c1c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2ffff700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c30074370 0x7f0c1c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2ffff700 1 -- 172.21.15.113:0/1690575181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c1c004d90 msgr2=0x7f0c1c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2ffff700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c1c004d90 0x7f0c1c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2ffff700 1 -- 172.21.15.113:0/1690575181 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1c0035b0 con 0x7f0c1c003a00 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c34e49700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c1c004d90 0x7f0c1c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.821+0000 7f0c2ffff700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c1c003a00 0x7f0c1c003e20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009b60 tx=0x7f0c1c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae67a013f0e141de server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.822+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1800e6c0 con 0x7f0c1c003a00 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.822+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1800e820 con 0x7f0c1c003a00 2024-01-15T11:26:15.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.822+0000 7f0c2effd700 1 -- 172.21.15.113:0/1690575181 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1800eb90 con 0x7f0c1c003a00 2024-01-15T11:26:15.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.822+0000 7f0c368ac700 1 -- 172.21.15.113:0/1690575181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c1c003a00 msgr2=0x7f0c1c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.822+0000 7f0c368ac700 1 --2- 172.21.15.113:0/1690575181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c1c003a00 0x7f0c1c003e20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009b60 tx=0x7f0c1c005880 comp rx=0 tx=0).stop 2024-01-15T11:26:15.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.822+0000 7f0c34e49700 1 -- 172.21.15.113:0/1690575181 reap_dead start 2024-01-15T11:26:15.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.822+0000 7f0c368ac700 1 -- 172.21.15.113:0/1690575181 shutdown_connections 2024-01-15T11:26:15.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.822+0000 7f0c368ac700 1 -- 172.21.15.113:0/1690575181 >> 172.21.15.113:0/1690575181 conn(0x7f0c300b3130 msgr2=0x7f0c300d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:15.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.822+0000 7f0c368ac700 1 -- 172.21.15.113:0/1690575181 shutdown_connections 2024-01-15T11:26:15.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.822+0000 7f0c368ac700 1 -- 172.21.15.113:0/1690575181 wait complete. 2024-01-15T11:26:15.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.823+0000 7f0c368ac700 1 Processor -- start 2024-01-15T11:26:15.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.824+0000 7f0c368ac700 1 -- start start 2024-01-15T11:26:15.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.824+0000 7f0c368ac700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c1c003a00 0x7f0c300cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.824+0000 7f0c368ac700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c300b5920 0x7f0c300c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.824+0000 7f0c368ac700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c3015b3d0 0x7f0c3006efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.824+0000 7f0c368ac700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c300cb360 con 0x7f0c3015b3d0 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.824+0000 7f0c368ac700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c300ca410 con 0x7f0c300b5920 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.824+0000 7f0c368ac700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c300ca6e0 con 0x7f0c1c003a00 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.824+0000 7f0c34e49700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c3015b3d0 0x7f0c3006efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.824+0000 7f0c34e49700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c3015b3d0 0x7f0c3006efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52022/0 (socket says 172.21.15.113:52022) 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.824+0000 7f0c34e49700 1 -- 172.21.15.113:0/48904803 learned_addr learned my addr 172.21.15.113:0/48904803 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c34e49700 1 -- 172.21.15.113:0/48904803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c1c003a00 msgr2=0x7f0c300cb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c2f7fe700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c300b5920 0x7f0c300c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c2ffff700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c1c003a00 0x7f0c300cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c34e49700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c1c003a00 0x7f0c300cb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c34e49700 1 -- 172.21.15.113:0/48904803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c300b5920 msgr2=0x7f0c300c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c34e49700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c300b5920 0x7f0c300c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c34e49700 1 -- 172.21.15.113:0/48904803 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c300ca9c0 con 0x7f0c3015b3d0 2024-01-15T11:26:15.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c34e49700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c3015b3d0 0x7f0c3006efe0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0c28000f80 tx=0x7f0c28007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:15.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c2cff9700 1 -- 172.21.15.113:0/48904803 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2800ee20 con 0x7f0c3015b3d0 2024-01-15T11:26:15.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c2cff9700 1 -- 172.21.15.113:0/48904803 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c28013420 con 0x7f0c3015b3d0 2024-01-15T11:26:15.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c2cff9700 1 -- 172.21.15.113:0/48904803 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c280137d0 con 0x7f0c3015b3d0 2024-01-15T11:26:15.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c368ac700 1 -- 172.21.15.113:0/48904803 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c300caca0 con 0x7f0c3015b3d0 2024-01-15T11:26:15.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.825+0000 7f0c368ac700 1 -- 172.21.15.113:0/48904803 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3015b850 con 0x7f0c3015b3d0 2024-01-15T11:26:15.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.826+0000 7f0c368ac700 1 -- 172.21.15.113:0/48904803 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c30067f60 con 0x7f0c3015b3d0 2024-01-15T11:26:15.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.826+0000 7f0c2cff9700 1 -- 172.21.15.113:0/48904803 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0c28013930 con 0x7f0c3015b3d0 2024-01-15T11:26:15.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.827+0000 7f0c2cff9700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0c1006b350 0x7f0c1006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:15.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.827+0000 7f0c2cff9700 1 -- 172.21.15.113:0/48904803 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c2808a290 con 0x7f0c3015b3d0 2024-01-15T11:26:15.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.827+0000 7f0c2ffff700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0c1006b350 0x7f0c1006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:15.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.827+0000 7f0c2ffff700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0c1006b350 0x7f0c1006d810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0c1800a010 tx=0x7f0c18008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:15.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:15.831+0000 7f0c2cff9700 1 -- 172.21.15.113:0/48904803 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c28055100 con 0x7f0c3015b3d0 2024-01-15T11:26:16.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.015+0000 7f0c368ac700 1 -- 172.21.15.113:0/48904803 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0c3015ba10 con 0x7f0c1006b350 2024-01-15T11:26:16.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.023+0000 7f0c2cff9700 1 -- 172.21.15.113:0/48904803 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289275 (secure 0 0 0) 0x7f0c3015ba10 con 0x7f0c1006b350 2024-01-15T11:26:16.025 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:16.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.026+0000 7f0c368ac700 1 -- 172.21.15.113:0/48904803 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0c1006b350 msgr2=0x7f0c1006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.026+0000 7f0c368ac700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0c1006b350 0x7f0c1006d810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0c1800a010 tx=0x7f0c18008030 comp rx=0 tx=0).stop 2024-01-15T11:26:16.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.026+0000 7f0c368ac700 1 -- 172.21.15.113:0/48904803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c3015b3d0 msgr2=0x7f0c3006efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.026+0000 7f0c368ac700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c3015b3d0 0x7f0c3006efe0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0c28000f80 tx=0x7f0c28007c90 comp rx=0 tx=0).stop 2024-01-15T11:26:16.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.026+0000 7f0c368ac700 1 -- 172.21.15.113:0/48904803 shutdown_connections 2024-01-15T11:26:16.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.026+0000 7f0c368ac700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f0c1006b350 0x7f0c1006d810 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.026+0000 7f0c368ac700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c3015b3d0 0x7f0c3006efe0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.026+0000 7f0c368ac700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0c1c003a00 0x7f0c300cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.026+0000 7f0c368ac700 1 --2- 172.21.15.113:0/48904803 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0c300b5920 0x7f0c300c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.026+0000 7f0c368ac700 1 -- 172.21.15.113:0/48904803 >> 172.21.15.113:0/48904803 conn(0x7f0c300b3130 msgr2=0x7f0c30144c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:16.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.026+0000 7f0c368ac700 1 -- 172.21.15.113:0/48904803 shutdown_connections 2024-01-15T11:26:16.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.026+0000 7f0c368ac700 1 -- 172.21.15.113:0/48904803 wait complete. 2024-01-15T11:26:16.029 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-15T11:26:16.037 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-01-15T11:26:14.120651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220144,"kb_used_data":3948,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754576,"statfs":{"total":773094113280,"available":772868685824,"internally_reserved":0,"allocated":4042752,"data_stored":2271487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.588869"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.567900+0000","last_change":"2024-01-15T11:26:11.564123+0000","last_active":"2024-01-15T11:26:11.567900+0000","last_peered":"2024-01-15T11:26:11.567900+0000","last_clean":"2024-01-15T11:26:11.567900+0000","last_became_active":"2024-01-15T11:26:11.563850+0000","last_became_peered":"2024-01-15T11:26:11.563850+0000","last_unstale":"2024-01-15T11:26:11.567900+0000","last_undegraded":"2024-01-15T11:26:11.567900+0000","last_fullsized":"2024-01-15T11:26:11.567900+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:39.545648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560581+0000","last_change":"2024-01-15T11:26:11.555663+0000","last_active":"2024-01-15T11:26:11.560581+0000","last_peered":"2024-01-15T11:26:11.560581+0000","last_clean":"2024-01-15T11:26:11.560581+0000","last_became_active":"2024-01-15T11:26:11.550879+0000","last_became_peered":"2024-01-15T11:26:11.550879+0000","last_unstale":"2024-01-15T11:26:11.560581+0000","last_undegraded":"2024-01-15T11:26:11.560581+0000","last_fullsized":"2024-01-15T11:26:11.560581+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:40:49.107290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.568044+0000","last_change":"2024-01-15T11:26:11.566884+0000","last_active":"2024-01-15T11:26:11.568044+0000","last_peered":"2024-01-15T11:26:11.568044+0000","last_clean":"2024-01-15T11:26:11.568044+0000","last_became_active":"2024-01-15T11:26:11.566612+0000","last_became_peered":"2024-01-15T11:26:11.566612+0000","last_unstale":"2024-01-15T11:26:11.568044+0000","last_undegraded":"2024-01-15T11:26:11.568044+0000","last_fullsized":"2024-01-15T11:26:11.568044+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:12.731404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.569507+0000","last_change":"2024-01-15T11:26:11.567887+0000","last_active":"2024-01-15T11:26:11.569507+0000","last_peered":"2024-01-15T11:26:11.569507+0000","last_clean":"2024-01-15T11:26:11.569507+0000","last_became_active":"2024-01-15T11:26:11.567624+0000","last_became_peered":"2024-01-15T11:26:11.567624+0000","last_unstale":"2024-01-15T11:26:11.569507+0000","last_undegraded":"2024-01-15T11:26:11.569507+0000","last_fullsized":"2024-01-15T11:26:11.569507+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:27:57.735875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565135+0000","last_change":"2024-01-15T11:26:11.561906+0000","last_active":"2024-01-15T11:26:11.565135+0000","last_peered":"2024-01-15T11:26:11.565135+0000","last_clean":"2024-01-15T11:26:11.565135+0000","last_became_active":"2024-01-15T11:26:11.561631+0000","last_became_peered":"2024-01-15T11:26:11.561631+0000","last_unstale":"2024-01-15T11:26:11.565135+0000","last_undegraded":"2024-01-15T11:26:11.565135+0000","last_fullsized":"2024-01-15T11:26:11.565135+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:47:02.390456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560691+0000","last_change":"2024-01-15T11:26:11.553208+0000","last_active":"2024-01-15T11:26:11.560691+0000","last_peered":"2024-01-15T11:26:11.560691+0000","last_clean":"2024-01-15T11:26:11.560691+0000","last_became_active":"2024-01-15T11:26:11.550184+0000","last_became_peered":"2024-01-15T11:26:11.550184+0000","last_unstale":"2024-01-15T11:26:11.560691+0000","last_undegraded":"2024-01-15T11:26:11.560691+0000","last_fullsized":"2024-01-15T11:26:11.560691+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:59:32.790611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564334+0000","last_change":"2024-01-15T11:26:11.562018+0000","last_active":"2024-01-15T11:26:11.564334+0000","last_peered":"2024-01-15T11:26:11.564334+0000","last_clean":"2024-01-15T11:26:11.564334+0000","last_became_active":"2024-01-15T11:26:11.561589+0000","last_became_peered":"2024-01-15T11:26:11.561589+0000","last_unstale":"2024-01-15T11:26:11.564334+0000","last_undegraded":"2024-01-15T11:26:11.564334+0000","last_fullsized":"2024-01-15T11:26:11.564334+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:09:11.710169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.563345+0000","last_change":"2024-01-15T11:26:11.559776+0000","last_active":"2024-01-15T11:26:11.563345+0000","last_peered":"2024-01-15T11:26:11.563345+0000","last_clean":"2024-01-15T11:26:11.563345+0000","last_became_active":"2024-01-15T11:26:11.559428+0000","last_became_peered":"2024-01-15T11:26:11.559428+0000","last_unstale":"2024-01-15T11:26:11.563345+0000","last_undegraded":"2024-01-15T11:26:11.563345+0000","last_fullsized":"2024-01-15T11:26:11.563345+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:54:28.575899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.569858+0000","last_change":"2024-01-15T11:26:11.568633+0000","last_active":"2024-01-15T11:26:11.569858+0000","last_peered":"2024-01-15T11:26:11.569858+0000","last_clean":"2024-01-15T11:26:11.569858+0000","last_became_active":"2024-01-15T11:26:11.568362+0000","last_became_peered":"2024-01-15T11:26:11.568362+0000","last_unstale":"2024-01-15T11:26:11.569858+0000","last_undegraded":"2024-01-15T11:26:11.569858+0000","last_fullsized":"2024-01-15T11:26:11.569858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:35:16.780032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566021+0000","last_change":"2024-01-15T11:26:11.561718+0000","last_active":"2024-01-15T11:26:11.566021+0000","last_peered":"2024-01-15T11:26:11.566021+0000","last_clean":"2024-01-15T11:26:11.566021+0000","last_became_active":"2024-01-15T11:26:11.561465+0000","last_became_peered":"2024-01-15T11:26:11.561465+0000","last_unstale":"2024-01-15T11:26:11.566021+0000","last_undegraded":"2024-01-15T11:26:11.566021+0000","last_fullsized":"2024-01-15T11:26:11.566021+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:39.545648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.568328+0000","last_change":"2024-01-15T11:26:11.565927+0000","last_active":"2024-01-15T11:26:11.568328+0000","last_peered":"2024-01-15T11:26:11.568328+0000","last_clean":"2024-01-15T11:26:11.568328+0000","last_became_active":"2024-01-15T11:26:11.565645+0000","last_became_peered":"2024-01-15T11:26:11.565645+0000","last_unstale":"2024-01-15T11:26:11.568328+0000","last_undegraded":"2024-01-15T11:26:11.568328+0000","last_fullsized":"2024-01-15T11:26:11.568328+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:12:09.208938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.551539+0000","last_change":"2024-01-15T11:26:11.546800+0000","last_active":"2024-01-15T11:26:11.551539+0000","last_peered":"2024-01-15T11:26:11.551539+0000","last_clean":"2024-01-15T11:26:11.551539+0000","last_became_active":"2024-01-15T11:26:11.546487+0000","last_became_peered":"2024-01-15T11:26:11.546487+0000","last_unstale":"2024-01-15T11:26:11.551539+0000","last_undegraded":"2024-01-15T11:26:11.551539+0000","last_fullsized":"2024-01-15T11:26:11.551539+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:52.795135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565154+0000","last_change":"2024-01-15T11:26:11.562595+0000","last_active":"2024-01-15T11:26:11.565154+0000","last_peered":"2024-01-15T11:26:11.565154+0000","last_clean":"2024-01-15T11:26:11.565154+0000","last_became_active":"2024-01-15T11:26:11.562326+0000","last_became_peered":"2024-01-15T11:26:11.562326+0000","last_unstale":"2024-01-15T11:26:11.565154+0000","last_undegraded":"2024-01-15T11:26:11.565154+0000","last_fullsized":"2024-01-15T11:26:11.565154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:38:46.949689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560735+0000","last_change":"2024-01-15T11:26:11.559121+0000","last_active":"2024-01-15T11:26:11.560735+0000","last_peered":"2024-01-15T11:26:11.560735+0000","last_clean":"2024-01-15T11:26:11.560735+0000","last_became_active":"2024-01-15T11:26:11.558880+0000","last_became_peered":"2024-01-15T11:26:11.558880+0000","last_unstale":"2024-01-15T11:26:11.560735+0000","last_undegraded":"2024-01-15T11:26:11.560735+0000","last_fullsized":"2024-01-15T11:26:11.560735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:51:14.891621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.563693+0000","last_change":"2024-01-15T11:26:11.560914+0000","last_active":"2024-01-15T11:26:11.563693+0000","last_peered":"2024-01-15T11:26:11.563693+0000","last_clean":"2024-01-15T11:26:11.563693+0000","last_became_active":"2024-01-15T11:26:11.560577+0000","last_became_peered":"2024-01-15T11:26:11.560577+0000","last_unstale":"2024-01-15T11:26:11.563693+0000","last_undegraded":"2024-01-15T11:26:11.563693+0000","last_fullsized":"2024-01-15T11:26:11.563693+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:26.983127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564293+0000","last_change":"2024-01-15T11:26:11.560881+0000","last_active":"2024-01-15T11:26:11.564293+0000","last_peered":"2024-01-15T11:26:11.564293+0000","last_clean":"2024-01-15T11:26:11.564293+0000","last_became_active":"2024-01-15T11:26:11.560569+0000","last_became_peered":"2024-01-15T11:26:11.560569+0000","last_unstale":"2024-01-15T11:26:11.564293+0000","last_undegraded":"2024-01-15T11:26:11.564293+0000","last_fullsized":"2024-01-15T11:26:11.564293+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:16.794640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.563321+0000","last_change":"2024-01-15T11:26:11.559811+0000","last_active":"2024-01-15T11:26:11.563321+0000","last_peered":"2024-01-15T11:26:11.563321+0000","last_clean":"2024-01-15T11:26:11.563321+0000","last_became_active":"2024-01-15T11:26:11.555585+0000","last_became_peered":"2024-01-15T11:26:11.555585+0000","last_unstale":"2024-01-15T11:26:11.563321+0000","last_undegraded":"2024-01-15T11:26:11.563321+0000","last_fullsized":"2024-01-15T11:26:11.563321+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:40.189089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.570449+0000","last_change":"2024-01-15T11:26:11.563623+0000","last_active":"2024-01-15T11:26:11.570449+0000","last_peered":"2024-01-15T11:26:11.570449+0000","last_clean":"2024-01-15T11:26:11.570449+0000","last_became_active":"2024-01-15T11:26:11.563335+0000","last_became_peered":"2024-01-15T11:26:11.563335+0000","last_unstale":"2024-01-15T11:26:11.570449+0000","last_undegraded":"2024-01-15T11:26:11.570449+0000","last_fullsized":"2024-01-15T11:26:11.570449+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:54:28.575899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.571068+0000","last_change":"2024-01-15T11:26:11.568691+0000","last_active":"2024-01-15T11:26:11.571068+0000","last_peered":"2024-01-15T11:26:11.571068+0000","last_clean":"2024-01-15T11:26:11.571068+0000","last_became_active":"2024-01-15T11:26:11.568396+0000","last_became_peered":"2024-01-15T11:26:11.568396+0000","last_unstale":"2024-01-15T11:26:11.571068+0000","last_undegraded":"2024-01-15T11:26:11.571068+0000","last_fullsized":"2024-01-15T11:26:11.571068+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:40.189089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564924+0000","last_change":"2024-01-15T11:26:11.558062+0000","last_active":"2024-01-15T11:26:11.564924+0000","last_peered":"2024-01-15T11:26:11.564924+0000","last_clean":"2024-01-15T11:26:11.564924+0000","last_became_active":"2024-01-15T11:26:11.557802+0000","last_became_peered":"2024-01-15T11:26:11.557802+0000","last_unstale":"2024-01-15T11:26:11.564924+0000","last_undegraded":"2024-01-15T11:26:11.564924+0000","last_fullsized":"2024-01-15T11:26:11.564924+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:33:18.117063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566567+0000","last_change":"2024-01-15T11:26:11.562143+0000","last_active":"2024-01-15T11:26:11.566567+0000","last_peered":"2024-01-15T11:26:11.566567+0000","last_clean":"2024-01-15T11:26:11.566567+0000","last_became_active":"2024-01-15T11:26:11.561707+0000","last_became_peered":"2024-01-15T11:26:11.561707+0000","last_unstale":"2024-01-15T11:26:11.566567+0000","last_undegraded":"2024-01-15T11:26:11.566567+0000","last_fullsized":"2024-01-15T11:26:11.566567+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:27:57.735875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566565+0000","last_change":"2024-01-15T11:26:11.562532+0000","last_active":"2024-01-15T11:26:11.566565+0000","last_peered":"2024-01-15T11:26:11.566565+0000","last_clean":"2024-01-15T11:26:11.566565+0000","last_became_active":"2024-01-15T11:26:11.562246+0000","last_became_peered":"2024-01-15T11:26:11.562246+0000","last_unstale":"2024-01-15T11:26:11.566565+0000","last_undegraded":"2024-01-15T11:26:11.566565+0000","last_fullsized":"2024-01-15T11:26:11.566565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:32:06.812304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.561304+0000","last_change":"2024-01-15T11:26:11.555289+0000","last_active":"2024-01-15T11:26:11.561304+0000","last_peered":"2024-01-15T11:26:11.561304+0000","last_clean":"2024-01-15T11:26:11.561304+0000","last_became_active":"2024-01-15T11:26:11.554832+0000","last_became_peered":"2024-01-15T11:26:11.554832+0000","last_unstale":"2024-01-15T11:26:11.561304+0000","last_undegraded":"2024-01-15T11:26:11.561304+0000","last_fullsized":"2024-01-15T11:26:11.561304+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:18.187321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.562549+0000","last_change":"2024-01-15T11:26:11.557553+0000","last_active":"2024-01-15T11:26:11.562549+0000","last_peered":"2024-01-15T11:26:11.562549+0000","last_clean":"2024-01-15T11:26:11.562549+0000","last_became_active":"2024-01-15T11:26:11.557206+0000","last_became_peered":"2024-01-15T11:26:11.557206+0000","last_unstale":"2024-01-15T11:26:11.562549+0000","last_undegraded":"2024-01-15T11:26:11.562549+0000","last_fullsized":"2024-01-15T11:26:11.562549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:09:11.710169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.567761+0000","last_change":"2024-01-15T11:26:11.560669+0000","last_active":"2024-01-15T11:26:11.567761+0000","last_peered":"2024-01-15T11:26:11.567761+0000","last_clean":"2024-01-15T11:26:11.567761+0000","last_became_active":"2024-01-15T11:26:11.560364+0000","last_became_peered":"2024-01-15T11:26:11.560364+0000","last_unstale":"2024-01-15T11:26:11.567761+0000","last_undegraded":"2024-01-15T11:26:11.567761+0000","last_fullsized":"2024-01-15T11:26:11.567761+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:04:01.689068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560610+0000","last_change":"2024-01-15T11:26:11.557494+0000","last_active":"2024-01-15T11:26:11.560610+0000","last_peered":"2024-01-15T11:26:11.560610+0000","last_clean":"2024-01-15T11:26:11.560610+0000","last_became_active":"2024-01-15T11:26:11.555709+0000","last_became_peered":"2024-01-15T11:26:11.555709+0000","last_unstale":"2024-01-15T11:26:11.560610+0000","last_undegraded":"2024-01-15T11:26:11.560610+0000","last_fullsized":"2024-01-15T11:26:11.560610+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:26.983127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.569547+0000","last_change":"2024-01-15T11:26:11.568325+0000","last_active":"2024-01-15T11:26:11.569547+0000","last_peered":"2024-01-15T11:26:11.569547+0000","last_clean":"2024-01-15T11:26:11.569547+0000","last_became_active":"2024-01-15T11:26:11.567570+0000","last_became_peered":"2024-01-15T11:26:11.567570+0000","last_unstale":"2024-01-15T11:26:11.569547+0000","last_undegraded":"2024-01-15T11:26:11.569547+0000","last_fullsized":"2024-01-15T11:26:11.569547+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:23:17.014644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.554443+0000","last_change":"2024-01-15T11:26:11.550678+0000","last_active":"2024-01-15T11:26:11.554443+0000","last_peered":"2024-01-15T11:26:11.554443+0000","last_clean":"2024-01-15T11:26:11.554443+0000","last_became_active":"2024-01-15T11:26:11.550389+0000","last_became_peered":"2024-01-15T11:26:11.550389+0000","last_unstale":"2024-01-15T11:26:11.554443+0000","last_undegraded":"2024-01-15T11:26:11.554443+0000","last_fullsized":"2024-01-15T11:26:11.554443+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:40.189089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566581+0000","last_change":"2024-01-15T11:26:11.562364+0000","last_active":"2024-01-15T11:26:11.566581+0000","last_peered":"2024-01-15T11:26:11.566581+0000","last_clean":"2024-01-15T11:26:11.566581+0000","last_became_active":"2024-01-15T11:26:11.562012+0000","last_became_peered":"2024-01-15T11:26:11.562012+0000","last_unstale":"2024-01-15T11:26:11.566581+0000","last_undegraded":"2024-01-15T11:26:11.566581+0000","last_fullsized":"2024-01-15T11:26:11.566581+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:33:35.657813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564263+0000","last_change":"2024-01-15T11:26:11.560923+0000","last_active":"2024-01-15T11:26:11.564263+0000","last_peered":"2024-01-15T11:26:11.564263+0000","last_clean":"2024-01-15T11:26:11.564263+0000","last_became_active":"2024-01-15T11:26:11.560655+0000","last_became_peered":"2024-01-15T11:26:11.560655+0000","last_unstale":"2024-01-15T11:26:11.564263+0000","last_undegraded":"2024-01-15T11:26:11.564263+0000","last_fullsized":"2024-01-15T11:26:11.564263+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:12.731404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.567637+0000","last_change":"2024-01-15T11:26:11.559945+0000","last_active":"2024-01-15T11:26:11.567637+0000","last_peered":"2024-01-15T11:26:11.567637+0000","last_clean":"2024-01-15T11:26:11.567637+0000","last_became_active":"2024-01-15T11:26:11.559320+0000","last_became_peered":"2024-01-15T11:26:11.559320+0000","last_unstale":"2024-01-15T11:26:11.567637+0000","last_undegraded":"2024-01-15T11:26:11.567637+0000","last_fullsized":"2024-01-15T11:26:11.567637+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:32:43.889552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.559627+0000","last_change":"2024-01-15T11:26:11.557039+0000","last_active":"2024-01-15T11:26:11.559627+0000","last_peered":"2024-01-15T11:26:11.559627+0000","last_clean":"2024-01-15T11:26:11.559627+0000","last_became_active":"2024-01-15T11:26:11.555633+0000","last_became_peered":"2024-01-15T11:26:11.555633+0000","last_unstale":"2024-01-15T11:26:11.559627+0000","last_undegraded":"2024-01-15T11:26:11.559627+0000","last_fullsized":"2024-01-15T11:26:11.559627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:02:39.352286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.562212+0000","last_change":"2024-01-15T11:26:11.555082+0000","last_active":"2024-01-15T11:26:11.562212+0000","last_peered":"2024-01-15T11:26:11.562212+0000","last_clean":"2024-01-15T11:26:11.562212+0000","last_became_active":"2024-01-15T11:26:11.554758+0000","last_became_peered":"2024-01-15T11:26:11.554758+0000","last_unstale":"2024-01-15T11:26:11.562212+0000","last_undegraded":"2024-01-15T11:26:11.562212+0000","last_fullsized":"2024-01-15T11:26:11.562212+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:14:25.674690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565225+0000","last_change":"2024-01-15T11:26:10.551768+0000","last_active":"2024-01-15T11:26:11.565225+0000","last_peered":"2024-01-15T11:26:11.565225+0000","last_clean":"2024-01-15T11:26:11.565225+0000","last_became_active":"2024-01-15T11:26:10.551469+0000","last_became_peered":"2024-01-15T11:26:10.551469+0000","last_unstale":"2024-01-15T11:26:11.565225+0000","last_undegraded":"2024-01-15T11:26:11.565225+0000","last_fullsized":"2024-01-15T11:26:11.565225+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:04:00.683556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.567512+0000","last_change":"2024-01-15T11:26:11.558952+0000","last_active":"2024-01-15T11:26:11.567512+0000","last_peered":"2024-01-15T11:26:11.567512+0000","last_clean":"2024-01-15T11:26:11.567512+0000","last_became_active":"2024-01-15T11:26:11.558659+0000","last_became_peered":"2024-01-15T11:26:11.558659+0000","last_unstale":"2024-01-15T11:26:11.567512+0000","last_undegraded":"2024-01-15T11:26:11.567512+0000","last_fullsized":"2024-01-15T11:26:11.567512+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:14:25.674690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.559457+0000","last_change":"2024-01-15T11:26:10.553845+0000","last_active":"2024-01-15T11:26:11.559457+0000","last_peered":"2024-01-15T11:26:11.559457+0000","last_clean":"2024-01-15T11:26:11.559457+0000","last_became_active":"2024-01-15T11:26:10.553580+0000","last_became_peered":"2024-01-15T11:26:10.553580+0000","last_unstale":"2024-01-15T11:26:11.559457+0000","last_undegraded":"2024-01-15T11:26:11.559457+0000","last_fullsized":"2024-01-15T11:26:11.559457+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:27:56.730362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566602+0000","last_change":"2024-01-15T11:26:11.562991+0000","last_active":"2024-01-15T11:26:11.566602+0000","last_peered":"2024-01-15T11:26:11.566602+0000","last_clean":"2024-01-15T11:26:11.566602+0000","last_became_active":"2024-01-15T11:26:11.562747+0000","last_became_peered":"2024-01-15T11:26:11.562747+0000","last_unstale":"2024-01-15T11:26:11.566602+0000","last_undegraded":"2024-01-15T11:26:11.566602+0000","last_fullsized":"2024-01-15T11:26:11.566602+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:10.490220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.568305+0000","last_change":"2024-01-15T11:26:10.564715+0000","last_active":"2024-01-15T11:26:11.568305+0000","last_peered":"2024-01-15T11:26:11.568305+0000","last_clean":"2024-01-15T11:26:11.568305+0000","last_became_active":"2024-01-15T11:26:10.564360+0000","last_became_peered":"2024-01-15T11:26:10.564360+0000","last_unstale":"2024-01-15T11:26:11.568305+0000","last_undegraded":"2024-01-15T11:26:11.568305+0000","last_fullsized":"2024-01-15T11:26:11.568305+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:15.789127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.569529+0000","last_change":"2024-01-15T11:26:11.567982+0000","last_active":"2024-01-15T11:26:11.569529+0000","last_peered":"2024-01-15T11:26:11.569529+0000","last_clean":"2024-01-15T11:26:11.569529+0000","last_became_active":"2024-01-15T11:26:11.567245+0000","last_became_peered":"2024-01-15T11:26:11.567245+0000","last_unstale":"2024-01-15T11:26:11.569529+0000","last_undegraded":"2024-01-15T11:26:11.569529+0000","last_fullsized":"2024-01-15T11:26:11.569529+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:10.490220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565680+0000","last_change":"2024-01-15T11:26:10.553909+0000","last_active":"2024-01-15T11:26:11.565680+0000","last_peered":"2024-01-15T11:26:11.565680+0000","last_clean":"2024-01-15T11:26:11.565680+0000","last_became_active":"2024-01-15T11:26:10.553692+0000","last_became_peered":"2024-01-15T11:26:10.553692+0000","last_unstale":"2024-01-15T11:26:11.565680+0000","last_undegraded":"2024-01-15T11:26:11.565680+0000","last_fullsized":"2024-01-15T11:26:11.565680+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:39.115189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.558414+0000","last_change":"2024-01-15T11:26:11.554416+0000","last_active":"2024-01-15T11:26:11.558414+0000","last_peered":"2024-01-15T11:26:11.558414+0000","last_clean":"2024-01-15T11:26:11.558414+0000","last_became_active":"2024-01-15T11:26:11.553609+0000","last_became_peered":"2024-01-15T11:26:11.553609+0000","last_unstale":"2024-01-15T11:26:11.558414+0000","last_undegraded":"2024-01-15T11:26:11.558414+0000","last_fullsized":"2024-01-15T11:26:11.558414+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:26.983127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.561951+0000","last_change":"2024-01-15T11:26:10.556678+0000","last_active":"2024-01-15T11:26:11.561951+0000","last_peered":"2024-01-15T11:26:11.561951+0000","last_clean":"2024-01-15T11:26:11.561951+0000","last_became_active":"2024-01-15T11:26:10.556409+0000","last_became_peered":"2024-01-15T11:26:10.556409+0000","last_unstale":"2024-01-15T11:26:11.561951+0000","last_undegraded":"2024-01-15T11:26:11.561951+0000","last_fullsized":"2024-01-15T11:26:11.561951+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:33:17.216079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.555396+0000","last_change":"2024-01-15T11:26:11.553706+0000","last_active":"2024-01-15T11:26:11.555396+0000","last_peered":"2024-01-15T11:26:11.555396+0000","last_clean":"2024-01-15T11:26:11.555396+0000","last_became_active":"2024-01-15T11:26:11.553339+0000","last_became_peered":"2024-01-15T11:26:11.553339+0000","last_unstale":"2024-01-15T11:26:11.555396+0000","last_undegraded":"2024-01-15T11:26:11.555396+0000","last_fullsized":"2024-01-15T11:26:11.555396+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:45:31.756657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.562018+0000","last_change":"2024-01-15T11:26:10.554498+0000","last_active":"2024-01-15T11:26:11.562018+0000","last_peered":"2024-01-15T11:26:11.562018+0000","last_clean":"2024-01-15T11:26:11.562018+0000","last_became_active":"2024-01-15T11:26:10.554138+0000","last_became_peered":"2024-01-15T11:26:10.554138+0000","last_unstale":"2024-01-15T11:26:11.562018+0000","last_undegraded":"2024-01-15T11:26:11.562018+0000","last_fullsized":"2024-01-15T11:26:11.562018+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:38.540135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566322+0000","last_change":"2024-01-15T11:26:11.560035+0000","last_active":"2024-01-15T11:26:11.566322+0000","last_peered":"2024-01-15T11:26:11.566322+0000","last_clean":"2024-01-15T11:26:11.566322+0000","last_became_active":"2024-01-15T11:26:11.559762+0000","last_became_peered":"2024-01-15T11:26:11.559762+0000","last_unstale":"2024-01-15T11:26:11.566322+0000","last_undegraded":"2024-01-15T11:26:11.566322+0000","last_fullsized":"2024-01-15T11:26:11.566322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:40.120702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.541452+0000","last_change":"2024-01-15T11:26:10.545920+0000","last_active":"2024-01-15T11:26:11.541452+0000","last_peered":"2024-01-15T11:26:11.541452+0000","last_clean":"2024-01-15T11:26:11.541452+0000","last_became_active":"2024-01-15T11:26:10.545643+0000","last_became_peered":"2024-01-15T11:26:10.545643+0000","last_unstale":"2024-01-15T11:26:11.541452+0000","last_undegraded":"2024-01-15T11:26:11.541452+0000","last_fullsized":"2024-01-15T11:26:11.541452+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T12:55:09.992378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565406+0000","last_change":"2024-01-15T11:26:11.557757+0000","last_active":"2024-01-15T11:26:11.565406+0000","last_peered":"2024-01-15T11:26:11.565406+0000","last_clean":"2024-01-15T11:26:11.565406+0000","last_became_active":"2024-01-15T11:26:11.557499+0000","last_became_peered":"2024-01-15T11:26:11.557499+0000","last_unstale":"2024-01-15T11:26:11.565406+0000","last_undegraded":"2024-01-15T11:26:11.565406+0000","last_fullsized":"2024-01-15T11:26:11.565406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:18.187321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.547816+0000","last_change":"2024-01-15T11:26:10.556424+0000","last_active":"2024-01-15T11:26:11.547816+0000","last_peered":"2024-01-15T11:26:11.547816+0000","last_clean":"2024-01-15T11:26:11.547816+0000","last_became_active":"2024-01-15T11:26:10.556144+0000","last_became_peered":"2024-01-15T11:26:10.556144+0000","last_unstale":"2024-01-15T11:26:11.547816+0000","last_undegraded":"2024-01-15T11:26:11.547816+0000","last_fullsized":"2024-01-15T11:26:11.547816+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:32:42.884039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.555259+0000","last_change":"2024-01-15T11:26:11.552356+0000","last_active":"2024-01-15T11:26:11.555259+0000","last_peered":"2024-01-15T11:26:11.555259+0000","last_clean":"2024-01-15T11:26:11.555259+0000","last_became_active":"2024-01-15T11:26:11.552051+0000","last_became_peered":"2024-01-15T11:26:11.552051+0000","last_unstale":"2024-01-15T11:26:11.555259+0000","last_undegraded":"2024-01-15T11:26:11.555259+0000","last_fullsized":"2024-01-15T11:26:11.555259+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:12:09.208938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.544001+0000","last_change":"2024-01-15T11:26:10.546384+0000","last_active":"2024-01-15T11:26:11.544001+0000","last_peered":"2024-01-15T11:26:11.544001+0000","last_clean":"2024-01-15T11:26:11.544001+0000","last_became_active":"2024-01-15T11:26:10.545705+0000","last_became_peered":"2024-01-15T11:26:10.545705+0000","last_unstale":"2024-01-15T11:26:11.544001+0000","last_undegraded":"2024-01-15T11:26:11.544001+0000","last_fullsized":"2024-01-15T11:26:11.544001+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:44:53.089530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.571064+0000","last_change":"2024-01-15T11:26:11.567130+0000","last_active":"2024-01-15T11:26:11.571064+0000","last_peered":"2024-01-15T11:26:11.571064+0000","last_clean":"2024-01-15T11:26:11.571064+0000","last_became_active":"2024-01-15T11:26:11.563343+0000","last_became_peered":"2024-01-15T11:26:11.563343+0000","last_unstale":"2024-01-15T11:26:11.571064+0000","last_undegraded":"2024-01-15T11:26:11.571064+0000","last_fullsized":"2024-01-15T11:26:11.571064+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:02:39.352286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.550077+0000","last_change":"2024-01-15T11:26:10.552843+0000","last_active":"2024-01-15T11:26:11.550077+0000","last_peered":"2024-01-15T11:26:11.550077+0000","last_clean":"2024-01-15T11:26:11.550077+0000","last_became_active":"2024-01-15T11:26:10.552556+0000","last_became_peered":"2024-01-15T11:26:10.552556+0000","last_unstale":"2024-01-15T11:26:11.550077+0000","last_undegraded":"2024-01-15T11:26:11.550077+0000","last_fullsized":"2024-01-15T11:26:11.550077+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:38.540135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560573+0000","last_change":"2024-01-15T11:26:11.555669+0000","last_active":"2024-01-15T11:26:11.560573+0000","last_peered":"2024-01-15T11:26:11.560573+0000","last_clean":"2024-01-15T11:26:11.560573+0000","last_became_active":"2024-01-15T11:26:11.555423+0000","last_became_peered":"2024-01-15T11:26:11.555423+0000","last_unstale":"2024-01-15T11:26:11.560573+0000","last_undegraded":"2024-01-15T11:26:11.560573+0000","last_fullsized":"2024-01-15T11:26:11.560573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:26:19.933700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.555760+0000","last_change":"2024-01-15T11:26:10.550026+0000","last_active":"2024-01-15T11:26:11.555760+0000","last_peered":"2024-01-15T11:26:11.555760+0000","last_clean":"2024-01-15T11:26:11.555760+0000","last_became_active":"2024-01-15T11:26:10.549672+0000","last_became_peered":"2024-01-15T11:26:10.549672+0000","last_unstale":"2024-01-15T11:26:11.555760+0000","last_undegraded":"2024-01-15T11:26:11.555760+0000","last_fullsized":"2024-01-15T11:26:11.555760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:54:27.570386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566817+0000","last_change":"2024-01-15T11:26:11.564314+0000","last_active":"2024-01-15T11:26:11.566817+0000","last_peered":"2024-01-15T11:26:11.566817+0000","last_clean":"2024-01-15T11:26:11.566817+0000","last_became_active":"2024-01-15T11:26:11.563976+0000","last_became_peered":"2024-01-15T11:26:11.563976+0000","last_unstale":"2024-01-15T11:26:11.566817+0000","last_undegraded":"2024-01-15T11:26:11.566817+0000","last_fullsized":"2024-01-15T11:26:11.566817+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:51:14.891621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.572289+0000","last_change":"2024-01-15T11:26:10.555654+0000","last_active":"2024-01-15T11:26:11.572289+0000","last_peered":"2024-01-15T11:26:11.572289+0000","last_clean":"2024-01-15T11:26:11.572289+0000","last_became_active":"2024-01-15T11:26:10.555265+0000","last_became_peered":"2024-01-15T11:26:10.555265+0000","last_unstale":"2024-01-15T11:26:11.572289+0000","last_undegraded":"2024-01-15T11:26:11.572289+0000","last_fullsized":"2024-01-15T11:26:11.572289+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:11.725891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565994+0000","last_change":"2024-01-15T11:26:11.558260+0000","last_active":"2024-01-15T11:26:11.565994+0000","last_peered":"2024-01-15T11:26:11.565994+0000","last_clean":"2024-01-15T11:26:11.565994+0000","last_became_active":"2024-01-15T11:26:11.558016+0000","last_became_peered":"2024-01-15T11:26:11.558016+0000","last_unstale":"2024-01-15T11:26:11.565994+0000","last_undegraded":"2024-01-15T11:26:11.565994+0000","last_fullsized":"2024-01-15T11:26:11.565994+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:34:27.955198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565586+0000","last_change":"2024-01-15T11:26:10.554161+0000","last_active":"2024-01-15T11:26:11.565586+0000","last_peered":"2024-01-15T11:26:11.565586+0000","last_clean":"2024-01-15T11:26:11.565586+0000","last_became_active":"2024-01-15T11:26:10.553835+0000","last_became_peered":"2024-01-15T11:26:10.553835+0000","last_unstale":"2024-01-15T11:26:11.565586+0000","last_undegraded":"2024-01-15T11:26:11.565586+0000","last_fullsized":"2024-01-15T11:26:11.565586+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T12:55:09.992378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560535+0000","last_change":"2024-01-15T11:26:11.553748+0000","last_active":"2024-01-15T11:26:11.560535+0000","last_peered":"2024-01-15T11:26:11.560535+0000","last_clean":"2024-01-15T11:26:11.560535+0000","last_became_active":"2024-01-15T11:26:11.553361+0000","last_became_peered":"2024-01-15T11:26:11.553361+0000","last_unstale":"2024-01-15T11:26:11.560535+0000","last_undegraded":"2024-01-15T11:26:11.560535+0000","last_fullsized":"2024-01-15T11:26:11.560535+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:12:09.208938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.554659+0000","last_change":"2024-01-15T11:26:10.556577+0000","last_active":"2024-01-15T11:26:11.554659+0000","last_peered":"2024-01-15T11:26:11.554659+0000","last_clean":"2024-01-15T11:26:11.554659+0000","last_became_active":"2024-01-15T11:26:10.556263+0000","last_became_peered":"2024-01-15T11:26:10.556263+0000","last_unstale":"2024-01-15T11:26:11.554659+0000","last_undegraded":"2024-01-15T11:26:11.554659+0000","last_fullsized":"2024-01-15T11:26:11.554659+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:33:17.216079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.559606+0000","last_change":"2024-01-15T11:26:11.554161+0000","last_active":"2024-01-15T11:26:11.559606+0000","last_peered":"2024-01-15T11:26:11.559606+0000","last_clean":"2024-01-15T11:26:11.559606+0000","last_became_active":"2024-01-15T11:26:11.553854+0000","last_became_peered":"2024-01-15T11:26:11.553854+0000","last_unstale":"2024-01-15T11:26:11.559606+0000","last_undegraded":"2024-01-15T11:26:11.559606+0000","last_fullsized":"2024-01-15T11:26:11.559606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:40:49.107290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.541347+0000","last_change":"2024-01-15T11:26:10.544207+0000","last_active":"2024-01-15T11:26:11.541347+0000","last_peered":"2024-01-15T11:26:11.541347+0000","last_clean":"2024-01-15T11:26:11.541347+0000","last_became_active":"2024-01-15T11:26:10.543937+0000","last_became_peered":"2024-01-15T11:26:10.543937+0000","last_unstale":"2024-01-15T11:26:11.541347+0000","last_undegraded":"2024-01-15T11:26:11.541347+0000","last_fullsized":"2024-01-15T11:26:11.541347+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:51.789623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.567775+0000","last_change":"2024-01-15T11:26:11.560138+0000","last_active":"2024-01-15T11:26:11.567775+0000","last_peered":"2024-01-15T11:26:11.567775+0000","last_clean":"2024-01-15T11:26:11.567775+0000","last_became_active":"2024-01-15T11:26:11.559673+0000","last_became_peered":"2024-01-15T11:26:11.559673+0000","last_unstale":"2024-01-15T11:26:11.567775+0000","last_undegraded":"2024-01-15T11:26:11.567775+0000","last_fullsized":"2024-01-15T11:26:11.567775+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:27:57.735875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.569532+0000","last_change":"2024-01-15T11:26:10.556115+0000","last_active":"2024-01-15T11:26:11.569532+0000","last_peered":"2024-01-15T11:26:11.569532+0000","last_clean":"2024-01-15T11:26:11.569532+0000","last_became_active":"2024-01-15T11:26:10.555211+0000","last_became_peered":"2024-01-15T11:26:10.555211+0000","last_unstale":"2024-01-15T11:26:11.569532+0000","last_undegraded":"2024-01-15T11:26:11.569532+0000","last_fullsized":"2024-01-15T11:26:11.569532+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:45:30.751144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.561580+0000","last_change":"2024-01-15T11:26:11.555885+0000","last_active":"2024-01-15T11:26:11.561580+0000","last_peered":"2024-01-15T11:26:11.561580+0000","last_clean":"2024-01-15T11:26:11.561580+0000","last_became_active":"2024-01-15T11:26:11.555621+0000","last_became_peered":"2024-01-15T11:26:11.555621+0000","last_unstale":"2024-01-15T11:26:11.561580+0000","last_undegraded":"2024-01-15T11:26:11.561580+0000","last_fullsized":"2024-01-15T11:26:11.561580+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:18.187321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.571242+0000","last_change":"2024-01-15T11:26:10.565356+0000","last_active":"2024-01-15T11:26:11.571242+0000","last_peered":"2024-01-15T11:26:11.571242+0000","last_clean":"2024-01-15T11:26:11.571242+0000","last_became_active":"2024-01-15T11:26:10.564922+0000","last_became_peered":"2024-01-15T11:26:10.564922+0000","last_unstale":"2024-01-15T11:26:11.571242+0000","last_undegraded":"2024-01-15T11:26:11.571242+0000","last_fullsized":"2024-01-15T11:26:11.571242+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:45:30.751144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.563546+0000","last_change":"2024-01-15T11:26:11.558248+0000","last_active":"2024-01-15T11:26:11.563546+0000","last_peered":"2024-01-15T11:26:11.563546+0000","last_clean":"2024-01-15T11:26:11.563546+0000","last_became_active":"2024-01-15T11:26:11.556305+0000","last_became_peered":"2024-01-15T11:26:11.556305+0000","last_unstale":"2024-01-15T11:26:11.563546+0000","last_undegraded":"2024-01-15T11:26:11.563546+0000","last_fullsized":"2024-01-15T11:26:11.563546+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:59:32.790611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560348+0000","last_change":"2024-01-15T11:26:10.555836+0000","last_active":"2024-01-15T11:26:11.560348+0000","last_peered":"2024-01-15T11:26:11.560348+0000","last_clean":"2024-01-15T11:26:11.560348+0000","last_became_active":"2024-01-15T11:26:10.555554+0000","last_became_peered":"2024-01-15T11:26:10.555554+0000","last_unstale":"2024-01-15T11:26:11.560348+0000","last_undegraded":"2024-01-15T11:26:11.560348+0000","last_fullsized":"2024-01-15T11:26:11.560348+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:23:16.009131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566043+0000","last_change":"2024-01-15T11:26:11.557802+0000","last_active":"2024-01-15T11:26:11.566043+0000","last_peered":"2024-01-15T11:26:11.566043+0000","last_clean":"2024-01-15T11:26:11.566043+0000","last_became_active":"2024-01-15T11:26:11.557536+0000","last_became_peered":"2024-01-15T11:26:11.557536+0000","last_unstale":"2024-01-15T11:26:11.566043+0000","last_undegraded":"2024-01-15T11:26:11.566043+0000","last_fullsized":"2024-01-15T11:26:11.566043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:32:43.889552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.563392+0000","last_change":"2024-01-15T11:26:10.555888+0000","last_active":"2024-01-15T11:26:11.563392+0000","last_peered":"2024-01-15T11:26:11.563392+0000","last_clean":"2024-01-15T11:26:11.563392+0000","last_became_active":"2024-01-15T11:26:10.555535+0000","last_became_peered":"2024-01-15T11:26:10.555535+0000","last_unstale":"2024-01-15T11:26:11.563392+0000","last_undegraded":"2024-01-15T11:26:11.563392+0000","last_fullsized":"2024-01-15T11:26:11.563392+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:04:00.683556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564926+0000","last_change":"2024-01-15T11:26:11.556887+0000","last_active":"2024-01-15T11:26:11.564926+0000","last_peered":"2024-01-15T11:26:11.564926+0000","last_clean":"2024-01-15T11:26:11.564926+0000","last_became_active":"2024-01-15T11:26:11.556494+0000","last_became_peered":"2024-01-15T11:26:11.556494+0000","last_unstale":"2024-01-15T11:26:11.564926+0000","last_undegraded":"2024-01-15T11:26:11.564926+0000","last_fullsized":"2024-01-15T11:26:11.564926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:29:24.215640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.552199+0000","last_change":"2024-01-15T11:26:10.554108+0000","last_active":"2024-01-15T11:26:11.552199+0000","last_peered":"2024-01-15T11:26:11.552199+0000","last_clean":"2024-01-15T11:26:11.552199+0000","last_became_active":"2024-01-15T11:26:10.553801+0000","last_became_peered":"2024-01-15T11:26:10.553801+0000","last_unstale":"2024-01-15T11:26:11.552199+0000","last_undegraded":"2024-01-15T11:26:11.552199+0000","last_fullsized":"2024-01-15T11:26:11.552199+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:26:18.928187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564871+0000","last_change":"2024-01-15T11:26:11.557103+0000","last_active":"2024-01-15T11:26:11.564871+0000","last_peered":"2024-01-15T11:26:11.564871+0000","last_clean":"2024-01-15T11:26:11.564871+0000","last_became_active":"2024-01-15T11:26:11.556759+0000","last_became_peered":"2024-01-15T11:26:11.556759+0000","last_unstale":"2024-01-15T11:26:11.564871+0000","last_undegraded":"2024-01-15T11:26:11.564871+0000","last_fullsized":"2024-01-15T11:26:11.564871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:48:00.776358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.563430+0000","last_change":"2024-01-15T11:26:10.553997+0000","last_active":"2024-01-15T11:26:11.563430+0000","last_peered":"2024-01-15T11:26:11.563430+0000","last_clean":"2024-01-15T11:26:11.563430+0000","last_became_active":"2024-01-15T11:26:10.553687+0000","last_became_peered":"2024-01-15T11:26:10.553687+0000","last_unstale":"2024-01-15T11:26:11.563430+0000","last_undegraded":"2024-01-15T11:26:11.563430+0000","last_fullsized":"2024-01-15T11:26:11.563430+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:23:16.009131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565169+0000","last_change":"2024-01-15T11:26:11.562925+0000","last_active":"2024-01-15T11:26:11.565169+0000","last_peered":"2024-01-15T11:26:11.565169+0000","last_clean":"2024-01-15T11:26:11.565169+0000","last_became_active":"2024-01-15T11:26:11.562651+0000","last_became_peered":"2024-01-15T11:26:11.562651+0000","last_unstale":"2024-01-15T11:26:11.565169+0000","last_undegraded":"2024-01-15T11:26:11.565169+0000","last_fullsized":"2024-01-15T11:26:11.565169+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:19:22.144552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.551014+0000","last_change":"2024-01-15T11:26:10.553741+0000","last_active":"2024-01-15T11:26:11.551014+0000","last_peered":"2024-01-15T11:26:11.551014+0000","last_clean":"2024-01-15T11:26:11.551014+0000","last_became_active":"2024-01-15T11:26:10.553432+0000","last_became_peered":"2024-01-15T11:26:10.553432+0000","last_unstale":"2024-01-15T11:26:11.551014+0000","last_undegraded":"2024-01-15T11:26:11.551014+0000","last_fullsized":"2024-01-15T11:26:11.551014+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:17.181809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.565262+0000","last_change":"2024-01-15T11:26:11.558534+0000","last_active":"2024-01-15T11:26:11.565262+0000","last_peered":"2024-01-15T11:26:11.565262+0000","last_clean":"2024-01-15T11:26:11.565262+0000","last_became_active":"2024-01-15T11:26:11.556712+0000","last_became_peered":"2024-01-15T11:26:11.556712+0000","last_unstale":"2024-01-15T11:26:11.565262+0000","last_undegraded":"2024-01-15T11:26:11.565262+0000","last_fullsized":"2024-01-15T11:26:11.565262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:59:32.655545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.546323+0000","last_change":"2024-01-15T11:26:10.556697+0000","last_active":"2024-01-15T11:26:11.546323+0000","last_peered":"2024-01-15T11:26:11.546323+0000","last_clean":"2024-01-15T11:26:11.546323+0000","last_became_active":"2024-01-15T11:26:10.555974+0000","last_became_peered":"2024-01-15T11:26:10.555974+0000","last_unstale":"2024-01-15T11:26:11.546323+0000","last_undegraded":"2024-01-15T11:26:11.546323+0000","last_fullsized":"2024-01-15T11:26:11.546323+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:14:24.669178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564316+0000","last_change":"2024-01-15T11:26:11.561846+0000","last_active":"2024-01-15T11:26:11.564316+0000","last_peered":"2024-01-15T11:26:11.564316+0000","last_clean":"2024-01-15T11:26:11.564316+0000","last_became_active":"2024-01-15T11:26:11.561245+0000","last_became_peered":"2024-01-15T11:26:11.561245+0000","last_unstale":"2024-01-15T11:26:11.564316+0000","last_undegraded":"2024-01-15T11:26:11.564316+0000","last_fullsized":"2024-01-15T11:26:11.564316+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:51:14.891621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.542692+0000","last_change":"2024-01-15T11:26:10.549311+0000","last_active":"2024-01-15T11:26:11.542692+0000","last_peered":"2024-01-15T11:26:11.542692+0000","last_clean":"2024-01-15T11:26:11.542692+0000","last_became_active":"2024-01-15T11:26:10.549026+0000","last_became_peered":"2024-01-15T11:26:10.549026+0000","last_unstale":"2024-01-15T11:26:11.542692+0000","last_undegraded":"2024-01-15T11:26:11.542692+0000","last_fullsized":"2024-01-15T11:26:11.542692+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:15.789127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564941+0000","last_change":"2024-01-15T11:26:11.557587+0000","last_active":"2024-01-15T11:26:11.564941+0000","last_peered":"2024-01-15T11:26:11.564941+0000","last_clean":"2024-01-15T11:26:11.564941+0000","last_became_active":"2024-01-15T11:26:11.556932+0000","last_became_peered":"2024-01-15T11:26:11.556932+0000","last_unstale":"2024-01-15T11:26:11.564941+0000","last_undegraded":"2024-01-15T11:26:11.564941+0000","last_fullsized":"2024-01-15T11:26:11.564941+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:23:24.437335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.569917+0000","last_change":"2024-01-15T11:26:10.565465+0000","last_active":"2024-01-15T11:26:11.569917+0000","last_peered":"2024-01-15T11:26:11.569917+0000","last_clean":"2024-01-15T11:26:11.569917+0000","last_became_active":"2024-01-15T11:26:10.565175+0000","last_became_peered":"2024-01-15T11:26:10.565175+0000","last_unstale":"2024-01-15T11:26:11.569917+0000","last_undegraded":"2024-01-15T11:26:11.569917+0000","last_fullsized":"2024-01-15T11:26:11.569917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:51.789623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566522+0000","last_change":"2024-01-15T11:26:11.564289+0000","last_active":"2024-01-15T11:26:11.566522+0000","last_peered":"2024-01-15T11:26:11.566522+0000","last_clean":"2024-01-15T11:26:11.566522+0000","last_became_active":"2024-01-15T11:26:11.564008+0000","last_became_peered":"2024-01-15T11:26:11.564008+0000","last_unstale":"2024-01-15T11:26:11.566522+0000","last_undegraded":"2024-01-15T11:26:11.566522+0000","last_fullsized":"2024-01-15T11:26:11.566522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T14:40:49.107290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.550217+0000","last_change":"2024-01-15T11:26:10.554135+0000","last_active":"2024-01-15T11:26:11.550217+0000","last_peered":"2024-01-15T11:26:11.550217+0000","last_clean":"2024-01-15T11:26:11.550217+0000","last_became_active":"2024-01-15T11:26:10.553831+0000","last_became_peered":"2024-01-15T11:26:10.553831+0000","last_unstale":"2024-01-15T11:26:11.550217+0000","last_undegraded":"2024-01-15T11:26:11.550217+0000","last_fullsized":"2024-01-15T11:26:11.550217+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:04:00.683556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566826+0000","last_change":"2024-01-15T11:26:11.565505+0000","last_active":"2024-01-15T11:26:11.566826+0000","last_peered":"2024-01-15T11:26:11.566826+0000","last_clean":"2024-01-15T11:26:11.566826+0000","last_became_active":"2024-01-15T11:26:11.565196+0000","last_became_peered":"2024-01-15T11:26:11.565196+0000","last_unstale":"2024-01-15T11:26:11.566826+0000","last_undegraded":"2024-01-15T11:26:11.566826+0000","last_fullsized":"2024-01-15T11:26:11.566826+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:34:27.955198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.555696+0000","last_change":"2024-01-15T11:26:10.555925+0000","last_active":"2024-01-15T11:26:11.555696+0000","last_peered":"2024-01-15T11:26:11.555696+0000","last_clean":"2024-01-15T11:26:11.555696+0000","last_became_active":"2024-01-15T11:26:10.555448+0000","last_became_peered":"2024-01-15T11:26:10.555448+0000","last_unstale":"2024-01-15T11:26:11.555696+0000","last_undegraded":"2024-01-15T11:26:11.555696+0000","last_fullsized":"2024-01-15T11:26:11.555696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:26:18.928187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.567620+0000","last_change":"2024-01-15T11:26:11.559204+0000","last_active":"2024-01-15T11:26:11.567620+0000","last_peered":"2024-01-15T11:26:11.567620+0000","last_clean":"2024-01-15T11:26:11.567620+0000","last_became_active":"2024-01-15T11:26:11.558824+0000","last_became_peered":"2024-01-15T11:26:11.558824+0000","last_unstale":"2024-01-15T11:26:11.567620+0000","last_undegraded":"2024-01-15T11:26:11.567620+0000","last_fullsized":"2024-01-15T11:26:11.567620+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:35:16.780032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.550825+0000","last_change":"2024-01-15T11:26:10.551338+0000","last_active":"2024-01-15T11:26:11.550825+0000","last_peered":"2024-01-15T11:26:11.550825+0000","last_clean":"2024-01-15T11:26:11.550825+0000","last_became_active":"2024-01-15T11:26:10.551039+0000","last_became_peered":"2024-01-15T11:26:10.551039+0000","last_unstale":"2024-01-15T11:26:11.550825+0000","last_undegraded":"2024-01-15T11:26:11.550825+0000","last_fullsized":"2024-01-15T11:26:11.550825+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:09.484707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.561094+0000","last_change":"2024-01-15T11:26:11.556659+0000","last_active":"2024-01-15T11:26:11.561094+0000","last_peered":"2024-01-15T11:26:11.561094+0000","last_clean":"2024-01-15T11:26:11.561094+0000","last_became_active":"2024-01-15T11:26:11.556404+0000","last_became_peered":"2024-01-15T11:26:11.556404+0000","last_unstale":"2024-01-15T11:26:11.561094+0000","last_undegraded":"2024-01-15T11:26:11.561094+0000","last_fullsized":"2024-01-15T11:26:11.561094+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:32:43.889552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.554738+0000","last_change":"2024-01-15T11:26:10.559269+0000","last_active":"2024-01-15T11:26:11.554738+0000","last_peered":"2024-01-15T11:26:11.554738+0000","last_clean":"2024-01-15T11:26:11.554738+0000","last_became_active":"2024-01-15T11:26:10.558981+0000","last_became_peered":"2024-01-15T11:26:10.558981+0000","last_unstale":"2024-01-15T11:26:11.554738+0000","last_undegraded":"2024-01-15T11:26:11.554738+0000","last_fullsized":"2024-01-15T11:26:11.554738+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T12:55:09.992378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.560672+0000","last_change":"2024-01-15T11:26:11.558476+0000","last_active":"2024-01-15T11:26:11.560672+0000","last_peered":"2024-01-15T11:26:11.560672+0000","last_clean":"2024-01-15T11:26:11.560672+0000","last_became_active":"2024-01-15T11:26:11.558222+0000","last_became_peered":"2024-01-15T11:26:11.558222+0000","last_unstale":"2024-01-15T11:26:11.560672+0000","last_undegraded":"2024-01-15T11:26:11.560672+0000","last_fullsized":"2024-01-15T11:26:11.560672+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:33:18.221592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.546480+0000","last_change":"2024-01-15T11:26:10.556277+0000","last_active":"2024-01-15T11:26:11.546480+0000","last_peered":"2024-01-15T11:26:11.546480+0000","last_clean":"2024-01-15T11:26:11.546480+0000","last_became_active":"2024-01-15T11:26:10.555654+0000","last_became_peered":"2024-01-15T11:26:10.555654+0000","last_unstale":"2024-01-15T11:26:11.546480+0000","last_undegraded":"2024-01-15T11:26:11.546480+0000","last_fullsized":"2024-01-15T11:26:11.546480+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:34:26.949685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.566511+0000","last_change":"2024-01-15T11:26:11.562207+0000","last_active":"2024-01-15T11:26:11.566511+0000","last_peered":"2024-01-15T11:26:11.566511+0000","last_clean":"2024-01-15T11:26:11.566511+0000","last_became_active":"2024-01-15T11:26:11.561896+0000","last_became_peered":"2024-01-15T11:26:11.561896+0000","last_unstale":"2024-01-15T11:26:11.566511+0000","last_undegraded":"2024-01-15T11:26:11.566511+0000","last_fullsized":"2024-01-15T11:26:11.566511+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:47:02.390456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.541680+0000","last_change":"2024-01-15T11:26:10.545067+0000","last_active":"2024-01-15T11:26:11.541680+0000","last_peered":"2024-01-15T11:26:11.541680+0000","last_clean":"2024-01-15T11:26:11.541680+0000","last_became_active":"2024-01-15T11:26:10.544775+0000","last_became_peered":"2024-01-15T11:26:10.544775+0000","last_unstale":"2024-01-15T11:26:11.541680+0000","last_undegraded":"2024-01-15T11:26:11.541680+0000","last_fullsized":"2024-01-15T11:26:11.541680+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T13:09:10.704656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.564176+0000","last_change":"2024-01-15T11:26:11.560649+0000","last_active":"2024-01-15T11:26:11.564176+0000","last_peered":"2024-01-15T11:26:11.564176+0000","last_clean":"2024-01-15T11:26:11.564176+0000","last_became_active":"2024-01-15T11:26:11.560057+0000","last_became_peered":"2024-01-15T11:26:11.560057+0000","last_unstale":"2024-01-15T11:26:11.564176+0000","last_undegraded":"2024-01-15T11:26:11.564176+0000","last_fullsized":"2024-01-15T11:26:11.564176+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:10.531362+0000","last_clean_scrub_stamp":"2024-01-15T11:26:10.531362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:59:32.790611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T11:26:11.551096+0000","last_change":"2024-01-15T11:26:10.557573+0000","last_active":"2024-01-15T11:26:11.551096+0000","last_peered":"2024-01-15T11:26:11.551096+0000","last_clean":"2024-01-15T11:26:11.551096+0000","last_became_active":"2024-01-15T11:26:10.557279+0000","last_became_peered":"2024-01-15T11:26:10.557279+0000","last_unstale":"2024-01-15T11:26:11.551096+0000","last_undegraded":"2024-01-15T11:26:11.551096+0000","last_fullsized":"2024-01-15T11:26:11.551096+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:26:09.525849+0000","last_clean_scrub_stamp":"2024-01-15T11:26:09.525849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T11:34:26.949685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-15T11:26:10.546618+0000","last_change":"2024-01-15T11:25:03.506154+0000","last_active":"2024-01-15T11:26:10.546618+0000","last_peered":"2024-01-15T11:26:10.546618+0000","last_clean":"2024-01-15T11:26:10.546618+0000","last_became_active":"2024-01-15T11:25:03.504996+0000","last_became_peered":"2024-01-15T11:25:03.504996+0000","last_unstale":"2024-01-15T11:26:10.546618+0000","last_undegraded":"2024-01-15T11:26:10.546618+0000","last_fullsized":"2024-01-15T11:26:10.546618+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T11:25:02.494136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T11:25:02.494136+0000","last_clean_scrub_stamp":"2024-01-15T11:25:02.494136+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:22:33.043449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":723877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":3.371,"5min":3.371,"15min":3.371},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":3.4420000000000002,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":0.53200000000000003}]},{"osd":1,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.0609999999999999,"5min":6.0609999999999999,"15min":6.0609999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":10.359,"5min":10.359,"15min":10.359},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.5190000000000001,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.56599999999999995}]},{"osd":3,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":10.523,"5min":10.523,"15min":10.523},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.133,"5min":1.133,"15min":1.133},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":10.497999999999999,"5min":10.497999999999999,"15min":10.497999999999999},"last":0.68999999999999995}]},{"osd":4,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.42299999999999999}]},{"osd":5,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.371},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.40100000000000002}]},{"osd":6,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.495}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":368640,"data_stored":140900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.59699999999999998}]},{"osd":1,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.64900000000000002}]},{"osd":2,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.51200000000000001}]},{"osd":3,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.624},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66600000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":141918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":4.0620000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.45400000000000001}]},{"osd":2,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.46,"5min":4.46,"15min":4.46},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.9470000000000001,"5min":3.9470000000000001,"15min":3.9470000000000001},"last":0.55100000000000005}]},{"osd":3,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.6799999999999997,"5min":4.6799999999999997,"15min":4.6799999999999997},"last":4.1299999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":4.7190000000000003}]},{"osd":4,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":4.3620000000000001,"5min":4.3620000000000001,"15min":4.3620000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.9940000000000002,"5min":3.9940000000000002,"15min":3.9940000000000002},"last":0.497}]},{"osd":5,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.3979999999999997,"5min":4.3979999999999997,"15min":4.3979999999999997},"last":0.69999999999999996}]},{"osd":6,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":4.4290000000000003,"5min":4.4290000000000003,"15min":4.4290000000000003},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":4.0259999999999998,"5min":4.0259999999999998,"15min":4.0259999999999998},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":1.012}]}]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":723877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"last":0.216},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.372}]},{"osd":2,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"last":0.252},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"last":0.32000000000000001}]},{"osd":4,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.67800000000000005}]},{"osd":6,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.52200000000000002}]},{"osd":7,"last update":"Mon Jan 15 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.57499999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.109,"5min":5.109,"15min":5.109},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.2380000000000004,"5min":5.2380000000000004,"15min":5.2380000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":3.5910000000000002,"5min":3.5910000000000002,"15min":3.5910000000000002},"last":0.41099999999999998}]},{"osd":3,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":5.2649999999999997,"5min":5.2649999999999997,"15min":5.2649999999999997},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":3.6720000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":0.63}]},{"osd":5,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":5.2110000000000003,"5min":5.2110000000000003,"15min":5.2110000000000003},"last":0.58199999999999996}]},{"osd":6,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.7000000000000002,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":5.1500000000000004,"5min":5.1500000000000004,"15min":5.1500000000000004},"last":0.622}]},{"osd":7,"last update":"Mon Jan 15 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.6349999999999998,"5min":3.6349999999999998,"15min":3.6349999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.61199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.33800000000000002}]},{"osd":1,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.307}]},{"osd":2,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.36199999999999999}]},{"osd":4,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.57799999999999996}]},{"osd":5,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.54400000000000004}]},{"osd":6,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon Jan 15 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.51100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574864,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":368640,"data_stored":140388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.70199999999999996}]},{"osd":1,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.58199999999999996}]},{"osd":2,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.621},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.68500000000000005}]},{"osd":3,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.67000000000000004}]},{"osd":5,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44600000000000001}]},{"osd":6,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.504},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.434}]},{"osd":7,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.53200000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.8620000000000001,"5min":2.8620000000000001,"15min":2.8620000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.1139999999999999,"5min":5.1139999999999999,"15min":5.1139999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":0.47099999999999997}]},{"osd":2,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":7.4589999999999996,"5min":7.4589999999999996,"15min":7.4589999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":5.0350000000000001,"5min":5.0350000000000001,"15min":5.0350000000000001},"last":0.436}]},{"osd":3,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0569999999999999,"5min":3.0569999999999999,"15min":3.0569999999999999},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.501},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.36299999999999999}]},{"osd":7,"last update":"Mon Jan 15 11:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T11:26:16.041 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T11:26:16.041 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T11:26:16.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.207+0000 7f23d8d88700 1 Processor -- start 2024-01-15T11:26:16.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.207+0000 7f23d8d88700 1 -- start start 2024-01-15T11:26:16.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.208+0000 7f23d8d88700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d415e280 0x7f23d40b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.208+0000 7f23d8d88700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f23d40b5220 0x7f23d406e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.208+0000 7f23d8d88700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d40b42a0 con 0x7f23d4073f90 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.208+0000 7f23d8d88700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d40af900 con 0x7f23d40b5220 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.208+0000 7f23d8d88700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d4074d30 con 0x7f23d415e280 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d2d9d700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f23d40b5220 0x7f23d406e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d259c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d415e280 0x7f23d40b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d2d9d700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f23d40b5220 0x7f23d406e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:43010/0 (socket says 172.21.15.113:43010) 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d259c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d415e280 0x7f23d40b3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:60194/0 (socket says 172.21.15.113:60194) 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d2d9d700 1 -- 172.21.15.113:0/100616918 learned_addr learned my addr 172.21.15.113:0/100616918 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d1d9b700 1 --1- 172.21.15.113:0/100616918 >> v1:172.21.15.141:6789/0 conn(0x7f23d4073f90 0x7f23d40b4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:43590/0 (socket says 172.21.15.113:43590) 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d2d9d700 1 -- 172.21.15.113:0/100616918 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d415e280 msgr2=0x7f23d40b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d2d9d700 1 --2- 172.21.15.113:0/100616918 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d415e280 0x7f23d40b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d2d9d700 1 -- 172.21.15.113:0/100616918 >> v1:172.21.15.141:6789/0 conn(0x7f23d4073f90 legacy=0x7f23d40b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d2d9d700 1 -- 172.21.15.113:0/100616918 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d4074490 con 0x7f23d40b5220 2024-01-15T11:26:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d2d9d700 1 --2- 172.21.15.113:0/100616918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d40b5220 0x7f23d406e220 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f23cc009e50 tx=0x7f23cc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=537790354f5c6c2e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:16.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d159a700 1 -- 172.21.15.113:0/100616918 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc00e070 con 0x7f23d40b5220 2024-01-15T11:26:16.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d159a700 1 -- 172.21.15.113:0/100616918 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23cc00b910 con 0x7f23d40b5220 2024-01-15T11:26:16.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.209+0000 7f23d159a700 1 -- 172.21.15.113:0/100616918 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc00bcc0 con 0x7f23d40b5220 2024-01-15T11:26:16.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.210+0000 7f23d8d88700 1 -- 172.21.15.113:0/100616918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d40b5220 msgr2=0x7f23d406e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.210+0000 7f23d8d88700 1 --2- 172.21.15.113:0/100616918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d40b5220 0x7f23d406e220 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f23cc009e50 tx=0x7f23cc00b330 comp rx=0 tx=0).stop 2024-01-15T11:26:16.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.210+0000 7f23d8d88700 1 -- 172.21.15.113:0/100616918 shutdown_connections 2024-01-15T11:26:16.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.210+0000 7f23d8d88700 1 --2- 172.21.15.113:0/100616918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d40b5220 0x7f23d406e220 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.210+0000 7f23d8d88700 1 --2- 172.21.15.113:0/100616918 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d415e280 0x7f23d40b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.210+0000 7f23d8d88700 1 -- 172.21.15.113:0/100616918 >> 172.21.15.113:0/100616918 conn(0x7f23d4072b20 msgr2=0x7f23d406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:16.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.210+0000 7f23d8d88700 1 -- 172.21.15.113:0/100616918 shutdown_connections 2024-01-15T11:26:16.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.210+0000 7f23d8d88700 1 -- 172.21.15.113:0/100616918 wait complete. 2024-01-15T11:26:16.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.211+0000 7f23d8d88700 1 Processor -- start 2024-01-15T11:26:16.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.211+0000 7f23d8d88700 1 -- start start 2024-01-15T11:26:16.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.212+0000 7f23d8d88700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d4073f90 0x7f23d4153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.212+0000 7f23d8d88700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d40b5220 0x7f23d4160a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.212+0000 7f23d8d88700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d4160090 0x7f23d4149830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.212+0000 7f23d8d88700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d4153b20 con 0x7f23d4073f90 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.212+0000 7f23d8d88700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d4154480 con 0x7f23d40b5220 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.212+0000 7f23d8d88700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d4160f60 con 0x7f23d4160090 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.212+0000 7f23d259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d4073f90 0x7f23d4153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.212+0000 7f23d259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d4073f90 0x7f23d4153f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52026/0 (socket says 172.21.15.113:52026) 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.212+0000 7f23d259c700 1 -- 172.21.15.113:0/1977385871 learned_addr learned my addr 172.21.15.113:0/1977385871 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.212+0000 7f23d2d9d700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d4160090 0x7f23d4149830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.213+0000 7f23d1d9b700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d40b5220 0x7f23d4160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.213+0000 7f23d259c700 1 -- 172.21.15.113:0/1977385871 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d4160090 msgr2=0x7f23d4149830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.213+0000 7f23d259c700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d4160090 0x7f23d4149830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.213+0000 7f23d259c700 1 -- 172.21.15.113:0/1977385871 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d40b5220 msgr2=0x7f23d4160a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.213+0000 7f23d259c700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d40b5220 0x7f23d4160a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.213+0000 7f23d259c700 1 -- 172.21.15.113:0/1977385871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc009b30 con 0x7f23d4073f90 2024-01-15T11:26:16.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.213+0000 7f23d259c700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d4073f90 0x7f23d4153f40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f23c800e870 tx=0x7f23c800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:16.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.213+0000 7f23c2ffd700 1 -- 172.21.15.113:0/1977385871 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c8009590 con 0x7f23d4073f90 2024-01-15T11:26:16.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.213+0000 7f23c2ffd700 1 -- 172.21.15.113:0/1977385871 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c800ee70 con 0x7f23d4073f90 2024-01-15T11:26:16.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.213+0000 7f23c2ffd700 1 -- 172.21.15.113:0/1977385871 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c800c9f0 con 0x7f23d4073f90 2024-01-15T11:26:16.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.213+0000 7f23d8d88700 1 -- 172.21.15.113:0/1977385871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d4153210 con 0x7f23d4073f90 2024-01-15T11:26:16.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.213+0000 7f23d8d88700 1 -- 172.21.15.113:0/1977385871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d4152c20 con 0x7f23d4073f90 2024-01-15T11:26:16.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.214+0000 7f23d8d88700 1 -- 172.21.15.113:0/1977385871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d4069020 con 0x7f23d4073f90 2024-01-15T11:26:16.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.214+0000 7f23c2ffd700 1 -- 172.21.15.113:0/1977385871 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f23c800cb50 con 0x7f23d4073f90 2024-01-15T11:26:16.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.215+0000 7f23c2ffd700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f23b406b350 0x7f23b406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.215+0000 7f23d1d9b700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f23b406b350 0x7f23b406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.215+0000 7f23c2ffd700 1 -- 172.21.15.113:0/1977385871 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23c8089b30 con 0x7f23d4073f90 2024-01-15T11:26:16.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.215+0000 7f23d1d9b700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f23b406b350 0x7f23b406d810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f23d41531c0 tx=0x7f23c4007d10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:16.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.219+0000 7f23c2ffd700 1 -- 172.21.15.113:0/1977385871 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23c80549a0 con 0x7f23d4073f90 2024-01-15T11:26:16.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.487+0000 7f23d8d88700 1 -- 172.21.15.113:0/1977385871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f23d4161710 con 0x7f23d4073f90 2024-01-15T11:26:16.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.490+0000 7f23c2ffd700 1 -- 172.21.15.113:0/1977385871 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f23c80210e0 con 0x7f23d4073f90 2024-01-15T11:26:16.489 INFO:teuthology.orchestra.run.smithi113.stdout:[client.0] 2024-01-15T11:26:16.489 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDhFaVl1ysgARAAgjFqY3kbPDbaQ0A6WxgMLQ== 2024-01-15T11:26:16.489 INFO:teuthology.orchestra.run.smithi113.stdout: caps mds = "allow" 2024-01-15T11:26:16.489 INFO:teuthology.orchestra.run.smithi113.stdout: caps mon = "allow r" 2024-01-15T11:26:16.489 INFO:teuthology.orchestra.run.smithi113.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T11:26:16.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.492+0000 7f23d8d88700 1 -- 172.21.15.113:0/1977385871 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f23b406b350 msgr2=0x7f23b406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.492+0000 7f23d8d88700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f23b406b350 0x7f23b406d810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f23d41531c0 tx=0x7f23c4007d10 comp rx=0 tx=0).stop 2024-01-15T11:26:16.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.492+0000 7f23d8d88700 1 -- 172.21.15.113:0/1977385871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d4073f90 msgr2=0x7f23d4153f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.492+0000 7f23d8d88700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d4073f90 0x7f23d4153f40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f23c800e870 tx=0x7f23c800ec30 comp rx=0 tx=0).stop 2024-01-15T11:26:16.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.493+0000 7f23d8d88700 1 -- 172.21.15.113:0/1977385871 shutdown_connections 2024-01-15T11:26:16.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.493+0000 7f23d8d88700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f23b406b350 0x7f23b406d810 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.493+0000 7f23d8d88700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f23d4160090 0x7f23d4149830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.493+0000 7f23d8d88700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d4073f90 0x7f23d4153f40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.493+0000 7f23d8d88700 1 --2- 172.21.15.113:0/1977385871 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f23d40b5220 0x7f23d4160a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.493+0000 7f23d8d88700 1 -- 172.21.15.113:0/1977385871 >> 172.21.15.113:0/1977385871 conn(0x7f23d4072b20 msgr2=0x7f23d4159ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:16.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.493+0000 7f23d8d88700 1 -- 172.21.15.113:0/1977385871 shutdown_connections 2024-01-15T11:26:16.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.493+0000 7f23d8d88700 1 -- 172.21.15.113:0/1977385871 wait complete. 2024-01-15T11:26:16.493 INFO:teuthology.orchestra.run.smithi113.stderr:updated caps for client.0 2024-01-15T11:26:16.504 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T11:26:16.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.628+0000 7f71f76a2700 1 Processor -- start 2024-01-15T11:26:16.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.628+0000 7f71f76a2700 1 -- start start 2024-01-15T11:26:16.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71f76a2700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f00ce090 0x7f71f006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71f76a2700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f006e7f0 con 0x7f71f015e2a0 2024-01-15T11:26:16.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71f76a2700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f006dec0 con 0x7f71f0073f90 2024-01-15T11:26:16.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71f76a2700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f006e1a0 con 0x7f71f00ce090 2024-01-15T11:26:16.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71f543e700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f71f015e2a0 0x7f71f006ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56482/0 (socket says 172.21.15.113:56482) 2024-01-15T11:26:16.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71f543e700 1 -- 172.21.15.113:0/1120133708 learned_addr learned my addr 172.21.15.113:0/1120133708 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71f4c3d700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f00ce090 0x7f71f006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993840359 0 0) 0x7f71f006e7f0 con 0x7f71f015e2a0 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71dc0035b0 con 0x7f71f015e2a0 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 478129726 0 0) 0x7f71dc0035b0 con 0x7f71f015e2a0 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f00ce090 msgr2=0x7f71f006ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f00ce090 0x7f71f006ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 >> v1:172.21.15.141:6789/0 conn(0x7f71f0073f90 legacy=0x7f71f00cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f006e480 con 0x7f71f015e2a0 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f71ec001670 con 0x7f71f015e2a0 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 >> v1:172.21.15.113:6789/0 conn(0x7f71f015e2a0 legacy=0x7f71f006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71dc003a30 0x7f71dc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71dc0043e0 0x7f71dc0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f00ce090 0x7f71dc005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f006e1a0 con 0x7f71dc0043e0 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc00ba10 con 0x7f71dc003a30 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.629+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc00bbd0 con 0x7f71f00ce090 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71f4c3d700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71dc0043e0 0x7f71dc0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71f543e700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71dc003a30 0x7f71dc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71f5c3f700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f00ce090 0x7f71dc005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71f4c3d700 1 -- 172.21.15.113:0/1120133708 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f00ce090 msgr2=0x7f71dc005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71f4c3d700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f00ce090 0x7f71dc005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71f4c3d700 1 -- 172.21.15.113:0/1120133708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71dc003a30 msgr2=0x7f71dc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71f4c3d700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71dc003a30 0x7f71dc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71f4c3d700 1 -- 172.21.15.113:0/1120133708 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc0035b0 con 0x7f71dc0043e0 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71f543e700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71dc003a30 0x7f71dc003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71f4c3d700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71dc0043e0 0x7f71dc0048b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f71e4009b60 tx=0x7f71dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feda142ec361ba23 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e400e6c0 con 0x7f71dc0043e0 2024-01-15T11:26:16.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e400e820 con 0x7f71dc0043e0 2024-01-15T11:26:16.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71e3fff700 1 -- 172.21.15.113:0/1120133708 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e400eb90 con 0x7f71dc0043e0 2024-01-15T11:26:16.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71f76a2700 1 -- 172.21.15.113:0/1120133708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71dc0043e0 msgr2=0x7f71dc0048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.630+0000 7f71f76a2700 1 --2- 172.21.15.113:0/1120133708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71dc0043e0 0x7f71dc0048b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f71e4009b60 tx=0x7f71dc005880 comp rx=0 tx=0).stop 2024-01-15T11:26:16.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.631+0000 7f71f5c3f700 1 -- 172.21.15.113:0/1120133708 reap_dead start 2024-01-15T11:26:16.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.631+0000 7f71f76a2700 1 -- 172.21.15.113:0/1120133708 shutdown_connections 2024-01-15T11:26:16.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.631+0000 7f71f76a2700 1 -- 172.21.15.113:0/1120133708 >> 172.21.15.113:0/1120133708 conn(0x7f71f00cd570 msgr2=0x7f71f00cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:16.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.631+0000 7f71f76a2700 1 -- 172.21.15.113:0/1120133708 shutdown_connections 2024-01-15T11:26:16.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.631+0000 7f71f76a2700 1 -- 172.21.15.113:0/1120133708 wait complete. 2024-01-15T11:26:16.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.632+0000 7f71f76a2700 1 Processor -- start 2024-01-15T11:26:16.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.632+0000 7f71f76a2700 1 -- start start 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f76a2700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71dc0043e0 0x7f71f01f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f76a2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71f015cca0 0x7f71f00cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f76a2700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f01f6fc0 0x7f71f00b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f76a2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f01f7e80 con 0x7f71f015cca0 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f76a2700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f01f78c0 con 0x7f71dc0043e0 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f76a2700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f01f7ba0 con 0x7f71f01f6fc0 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f4c3d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71f015cca0 0x7f71f00cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f4c3d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71f015cca0 0x7f71f00cc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52038/0 (socket says 172.21.15.113:52038) 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f4c3d700 1 -- 172.21.15.113:0/1195784741 learned_addr learned my addr 172.21.15.113:0/1195784741 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f5c3f700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f01f6fc0 0x7f71f00b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f543e700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71dc0043e0 0x7f71f01f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f4c3d700 1 -- 172.21.15.113:0/1195784741 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f01f6fc0 msgr2=0x7f71f00b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f4c3d700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f01f6fc0 0x7f71f00b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f4c3d700 1 -- 172.21.15.113:0/1195784741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71dc0043e0 msgr2=0x7f71f01f4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f4c3d700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71dc0043e0 0x7f71f01f4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f4c3d700 1 -- 172.21.15.113:0/1195784741 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f01f3b70 con 0x7f71f015cca0 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f4c3d700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71f015cca0 0x7f71f00cc2e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f71e400e4b0 tx=0x7f71e40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71e1ffb700 1 -- 172.21.15.113:0/1195784741 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e400e6c0 con 0x7f71f015cca0 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71e1ffb700 1 -- 172.21.15.113:0/1195784741 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e4003ab0 con 0x7f71f015cca0 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f76a2700 1 -- 172.21.15.113:0/1195784741 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f01f3e50 con 0x7f71f015cca0 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71e1ffb700 1 -- 172.21.15.113:0/1195784741 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4003e60 con 0x7f71f015cca0 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.633+0000 7f71f76a2700 1 -- 172.21.15.113:0/1195784741 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f00cc820 con 0x7f71f015cca0 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.634+0000 7f71f76a2700 1 -- 172.21.15.113:0/1195784741 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f01f39d0 con 0x7f71f015cca0 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.635+0000 7f71e1ffb700 1 -- 172.21.15.113:0/1195784741 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f71e4013470 con 0x7f71f015cca0 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.635+0000 7f71e1ffb700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f71d406b350 0x7f71d406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.635+0000 7f71f543e700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f71d406b350 0x7f71d406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.635+0000 7f71e1ffb700 1 -- 172.21.15.113:0/1195784741 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71e408b550 con 0x7f71f015cca0 2024-01-15T11:26:16.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.636+0000 7f71f543e700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f71d406b350 0x7f71d406d810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f71f01606b0 tx=0x7f71ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:16.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.640+0000 7f71e1ffb700 1 -- 172.21.15.113:0/1195784741 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f71f01f39d0 con 0x7f71f015cca0 2024-01-15T11:26:16.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.873+0000 7f71f76a2700 1 -- 172.21.15.113:0/1195784741 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71f01f39d0 con 0x7f71f015cca0 2024-01-15T11:26:16.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.874+0000 7f71e1ffb700 1 -- 172.21.15.113:0/1195784741 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f71f01f39d0 con 0x7f71f015cca0 2024-01-15T11:26:16.873 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:16.873 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/1602579902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":1602579902},{"type":"v1","addr":"172.21.15.113:6836","nonce":1602579902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4915,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6834/3965637007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3965637007},{"type":"v1","addr":"172.21.15.141:6834","nonce":3965637007}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4918,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/4263133094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6833","nonce":4263133094},{"type":"v1","addr":"172.21.15.141:6835","nonce":4263133094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T11:26:10.938493+0000","modified":"2024-01-15T11:26:13.970629+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5004":{"gid":5004,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6837/436972884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":436972884},{"type":"v1","addr":"172.21.15.113:6837","nonce":436972884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-15T11:26:16.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.876+0000 7f71f76a2700 1 -- 172.21.15.113:0/1195784741 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f71d406b350 msgr2=0x7f71d406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.876+0000 7f71f76a2700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f71d406b350 0x7f71d406d810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f71f01606b0 tx=0x7f71ec009040 comp rx=0 tx=0).stop 2024-01-15T11:26:16.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.876+0000 7f71f76a2700 1 -- 172.21.15.113:0/1195784741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71f015cca0 msgr2=0x7f71f00cc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:16.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.876+0000 7f71f76a2700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71f015cca0 0x7f71f00cc2e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f71e400e4b0 tx=0x7f71e40047c0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.876+0000 7f71f76a2700 1 -- 172.21.15.113:0/1195784741 shutdown_connections 2024-01-15T11:26:16.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.876+0000 7f71f76a2700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f71d406b350 0x7f71d406d810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.876+0000 7f71f76a2700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f71f01f6fc0 0x7f71f00b4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.876+0000 7f71f76a2700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71dc0043e0 0x7f71f01f4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.876+0000 7f71f76a2700 1 --2- 172.21.15.113:0/1195784741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71f015cca0 0x7f71f00cc2e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:16.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.876+0000 7f71f76a2700 1 -- 172.21.15.113:0/1195784741 >> 172.21.15.113:0/1195784741 conn(0x7f71f00cd570 msgr2=0x7f71f00cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:16.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.876+0000 7f71f76a2700 1 -- 172.21.15.113:0/1195784741 shutdown_connections 2024-01-15T11:26:16.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:16.876+0000 7f71f76a2700 1 -- 172.21.15.113:0/1195784741 wait complete. 2024-01-15T11:26:16.876 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 17 2024-01-15T11:26:17.289 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T11:26:10.938493+0000', 'modified': '2024-01-15T11:26:13.970629+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5004}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5004': {'gid': 5004, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.113:6837/436972884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6835', 'nonce': 436972884}, {'type': 'v1', 'addr': '172.21.15.113:6837', 'nonce': 436972884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T11:26:17.289 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T11:26:17.290 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-15T11:26:17.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.416+0000 7ff6e757b700 1 Processor -- start 2024-01-15T11:26:17.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.416+0000 7ff6e757b700 1 -- start start 2024-01-15T11:26:17.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.417+0000 7ff6e757b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e006c470 0x7ff6e00b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.417+0000 7ff6e757b700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e006b3d0 con 0x7ff6e0073f90 2024-01-15T11:26:17.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.417+0000 7ff6e757b700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e006b6b0 con 0x7ff6e00b42d0 2024-01-15T11:26:17.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.417+0000 7ff6e757b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e006b990 con 0x7ff6e006c470 2024-01-15T11:26:17.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.417+0000 7ff6e5317700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ff6e0073f90 0x7ff6e0074370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56494/0 (socket says 172.21.15.113:56494) 2024-01-15T11:26:17.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.417+0000 7ff6e5317700 1 -- 172.21.15.113:0/4227826969 learned_addr learned my addr 172.21.15.113:0/4227826969 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:17.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294035960 0 0) 0x7ff6e006b3d0 con 0x7ff6e0073f90 2024-01-15T11:26:17.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6cc0035b0 con 0x7ff6e0073f90 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6e4b16700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e006c470 0x7ff6e00b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4207453587 0 0) 0x7ff6cc0035b0 con 0x7ff6e0073f90 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e006c470 msgr2=0x7ff6e00b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e006c470 0x7ff6e00b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 >> v1:172.21.15.141:6789/0 conn(0x7ff6e00b42d0 legacy=0x7ff6e006dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e006bc70 con 0x7ff6e0073f90 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7ff6d0001670 con 0x7ff6e0073f90 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 >> v1:172.21.15.113:6789/0 conn(0x7ff6e0073f90 legacy=0x7ff6e0074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6cc003a30 0x7ff6cc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6cc004850 0x7ff6cc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e006c470 0x7ff6cc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e006b990 con 0x7ff6cc003a30 2024-01-15T11:26:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc00bac0 con 0x7ff6cc004850 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc00bc80 con 0x7ff6e006c470 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6e5317700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6cc003a30 0x7ff6cc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6e4b16700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6cc004850 0x7ff6cc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e006c470 0x7ff6cc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6e5317700 1 -- 172.21.15.113:0/4227826969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e006c470 msgr2=0x7ff6cc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6e5317700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e006c470 0x7ff6cc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6e5317700 1 -- 172.21.15.113:0/4227826969 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6cc004850 msgr2=0x7ff6cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6e5317700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6cc004850 0x7ff6cc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6e5317700 1 -- 172.21.15.113:0/4227826969 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc0035b0 con 0x7ff6cc003a30 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.418+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e006c470 0x7ff6cc005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.419+0000 7ff6e5317700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6cc003a30 0x7ff6cc003e50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000c210 tx=0x7ff6cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99af7faa3b118ff9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.419+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0007c10 con 0x7ff6cc003a30 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.419+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d000cdd0 con 0x7ff6cc003a30 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.419+0000 7ff6d7fff700 1 -- 172.21.15.113:0/4227826969 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0005ba0 con 0x7ff6cc003a30 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.419+0000 7ff6e757b700 1 -- 172.21.15.113:0/4227826969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6cc003a30 msgr2=0x7ff6cc003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.419+0000 7ff6e757b700 1 --2- 172.21.15.113:0/4227826969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6cc003a30 0x7ff6cc003e50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000c210 tx=0x7ff6cc005880 comp rx=0 tx=0).stop 2024-01-15T11:26:17.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.419+0000 7ff6e5b18700 1 -- 172.21.15.113:0/4227826969 reap_dead start 2024-01-15T11:26:17.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.419+0000 7ff6e757b700 1 -- 172.21.15.113:0/4227826969 shutdown_connections 2024-01-15T11:26:17.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.419+0000 7ff6e757b700 1 -- 172.21.15.113:0/4227826969 >> 172.21.15.113:0/4227826969 conn(0x7ff6e00b3130 msgr2=0x7ff6e00d0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:17.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.419+0000 7ff6e757b700 1 -- 172.21.15.113:0/4227826969 shutdown_connections 2024-01-15T11:26:17.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.419+0000 7ff6e757b700 1 -- 172.21.15.113:0/4227826969 wait complete. 2024-01-15T11:26:17.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.420+0000 7ff6e757b700 1 Processor -- start 2024-01-15T11:26:17.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.421+0000 7ff6e757b700 1 -- start start 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.421+0000 7ff6e757b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6cc003a30 0x7ff6e00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.421+0000 7ff6e757b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e015da50 0x7ff6e0144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.421+0000 7ff6e757b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e00cef70 0x7ff6e00ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.421+0000 7ff6e757b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e0144a70 con 0x7ff6e015da50 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.421+0000 7ff6e757b700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e0143520 con 0x7ff6cc003a30 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.421+0000 7ff6e757b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e0143770 con 0x7ff6e00cef70 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.421+0000 7ff6e4b16700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e015da50 0x7ff6e0144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.421+0000 7ff6e4b16700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e015da50 0x7ff6e0144530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52062/0 (socket says 172.21.15.113:52062) 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.421+0000 7ff6e4b16700 1 -- 172.21.15.113:0/2002513662 learned_addr learned my addr 172.21.15.113:0/2002513662 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6e5317700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6cc003a30 0x7ff6e00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6e4b16700 1 -- 172.21.15.113:0/2002513662 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e00cef70 msgr2=0x7ff6e00ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e00cef70 0x7ff6e00ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6e4b16700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e00cef70 0x7ff6e00ae9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6e4b16700 1 -- 172.21.15.113:0/2002513662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6cc003a30 msgr2=0x7ff6e00cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6e4b16700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6cc003a30 0x7ff6e00cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6e4b16700 1 -- 172.21.15.113:0/2002513662 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e0143a50 con 0x7ff6e015da50 2024-01-15T11:26:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6e4b16700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e015da50 0x7ff6e0144530 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0119c0 tx=0x7ff6dc011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6d5ffb700 1 -- 172.21.15.113:0/2002513662 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc004240 con 0x7ff6e015da50 2024-01-15T11:26:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6d5ffb700 1 -- 172.21.15.113:0/2002513662 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc0043a0 con 0x7ff6e015da50 2024-01-15T11:26:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6e757b700 1 -- 172.21.15.113:0/2002513662 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e0143d30 con 0x7ff6e015da50 2024-01-15T11:26:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6d5ffb700 1 -- 172.21.15.113:0/2002513662 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0036a0 con 0x7ff6e015da50 2024-01-15T11:26:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.422+0000 7ff6e757b700 1 -- 172.21.15.113:0/2002513662 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e00cf3f0 con 0x7ff6e015da50 2024-01-15T11:26:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.424+0000 7ff6d5ffb700 1 -- 172.21.15.113:0/2002513662 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff6dc003800 con 0x7ff6e015da50 2024-01-15T11:26:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.424+0000 7ff6d5ffb700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff6c406b350 0x7ff6c406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.424+0000 7ff6d5ffb700 1 -- 172.21.15.113:0/2002513662 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6dc089f60 con 0x7ff6e015da50 2024-01-15T11:26:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.424+0000 7ff6e5317700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff6c406b350 0x7ff6c406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.425+0000 7ff6e5317700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff6c406b350 0x7ff6c406d810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0015040 tx=0x7ff6d0007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:17.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.426+0000 7ff6f3777b80 1 Processor -- start 2024-01-15T11:26:17.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.426+0000 7ff6f3777b80 1 -- start start 2024-01-15T11:26:17.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6f3777b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b1fce0 0x557a76bdacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6f3777b80 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76bbc410 0x557a76bb7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6f3777b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 0x557a76b941d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6f3777b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a76b07480 con 0x557a76b91dd0 2024-01-15T11:26:17.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6f3777b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a76aed770 con 0x557a76b1fce0 2024-01-15T11:26:17.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6f3777b80 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a76aeb090 con 0x557a76bbc410 2024-01-15T11:26:17.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6e5b18700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 0x557a76b941d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6e5b18700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 0x557a76b941d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52068/0 (socket says 172.21.15.113:52068) 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6e5b18700 1 -- 172.21.15.113:0/2135201578 learned_addr learned my addr 172.21.15.113:0/2135201578 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6e5317700 1 --2- 172.21.15.113:0/2135201578 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b1fce0 0x557a76bdacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6e4b16700 1 --2- 172.21.15.113:0/2135201578 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76bbc410 0x557a76bb7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6e5b18700 1 -- 172.21.15.113:0/2135201578 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76bbc410 msgr2=0x557a76bb7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/2135201578 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76bbc410 0x557a76bb7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6e5b18700 1 -- 172.21.15.113:0/2135201578 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b1fce0 msgr2=0x557a76bdacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/2135201578 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b1fce0 0x557a76bdacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6e5b18700 1 -- 172.21.15.113:0/2135201578 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a7690fb10 con 0x557a76b91dd0 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/2135201578 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 0x557a76b941d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0147ce0 tx=0x7ff6d8002150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b7f283c44595751 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6caffd700 1 -- 172.21.15.113:0/2135201578 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8003c10 con 0x557a76b91dd0 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6caffd700 1 -- 172.21.15.113:0/2135201578 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d800d050 con 0x557a76b91dd0 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.427+0000 7ff6caffd700 1 -- 172.21.15.113:0/2135201578 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d800f800 con 0x557a76b91dd0 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.428+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2135201578 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 msgr2=0x557a76b941d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.428+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2135201578 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 0x557a76b941d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0147ce0 tx=0x7ff6d8002150 comp rx=0 tx=0).stop 2024-01-15T11:26:17.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.428+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2135201578 shutdown_connections 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.428+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2135201578 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 0x557a76b941d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.428+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2135201578 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b1fce0 0x557a76bdacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.428+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2135201578 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76bbc410 0x557a76bb7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.428+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2135201578 >> 172.21.15.113:0/2135201578 conn(0x557a76ac1410 msgr2=0x557a76bd9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.428+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2135201578 shutdown_connections 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.428+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2135201578 wait complete. 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.429+0000 7ff6f3777b80 1 -- start start 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.429+0000 7ff6f3777b80 1 Processor -- start 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6f3777b80 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b1fce0 0x557a76bd36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6f3777b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 0x557a76bd0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6f3777b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76bbc410 0x557a76be9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6f3777b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a76ae6980 con 0x557a76b91dd0 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6f3777b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a76a86000 con 0x557a76bbc410 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6f3777b80 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a76a85ca0 con 0x557a76b1fce0 2024-01-15T11:26:17.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6f3777b80 10 client.0 _ll_register_callbacks cb 0x557a768e9350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6e4b16700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 0x557a76bd0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6e4b16700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 0x557a76bd0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52076/0 (socket says 172.21.15.113:52076) 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6e4b16700 1 -- 172.21.15.113:0/2791285718 learned_addr learned my addr 172.21.15.113:0/2791285718 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6e5317700 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b1fce0 0x557a76bd36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76bbc410 0x557a76be9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6e4b16700 1 -- 172.21.15.113:0/2791285718 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b1fce0 msgr2=0x557a76bd36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6e4b16700 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b1fce0 0x557a76bd36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6e4b16700 1 -- 172.21.15.113:0/2791285718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76bbc410 msgr2=0x557a76be9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6e4b16700 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76bbc410 0x557a76be9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6e4b16700 1 -- 172.21.15.113:0/2791285718 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e006b6b0 con 0x557a76b91dd0 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6e4b16700 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 0x557a76bd0870 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00f520 tx=0x7ff6dc058ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e9e012bf4469e3f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6bffff700 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6bffff700 1 -- 172.21.15.113:0/2791285718 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc060b90 con 0x557a76b91dd0 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6bffff700 1 -- 172.21.15.113:0/2791285718 <== mon.0 v2:172.21.15.113:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6dc060e20 con 0x557a76b91dd0 2024-01-15T11:26:17.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2791285718 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a76b06450 con 0x557a76b91dd0 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6bffff700 1 -- 172.21.15.113:0/2791285718 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6dc088080 con 0x557a76b91dd0 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.430+0000 7ff6f3777b80 10 client.5178 fetch_fsmap learned FSMap version 17 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.431+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2791285718 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x557a76aa8290 con 0x557a76b91dd0 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.431+0000 7ff6bffff700 1 -- 172.21.15.113:0/2791285718 <== mon.0 v2:172.21.15.113:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc074070 con 0x557a76b91dd0 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.431+0000 7ff6bffff700 1 -- 172.21.15.113:0/2791285718 <== mon.0 v2:172.21.15.113:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff6dc070180 con 0x557a76b91dd0 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.431+0000 7ff6f3777b80 10 client.5178 fetch_fsmap finished waiting for FSMap version 17 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.431+0000 7ff6f3777b80 10 client.5178 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.431+0000 7ff6f3777b80 20 client.5178 populate_metadata read hostname 'smithi113' 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.431+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.113:6835/436972884,v1:172.21.15.113:6837/436972884] conn(0x557a76b94060 0x557a76b921b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.431+0000 7ff6f3777b80 4 client.5178 mds_command: new command op to 5004 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.431+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2791285718 --> [v2:172.21.15.113:6835/436972884,v1:172.21.15.113:6837/436972884] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a76adf960 con 0x557a76b94060 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.431+0000 7ff6e5317700 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.113:6835/436972884,v1:172.21.15.113:6837/436972884] conn(0x557a76b94060 0x557a76b921b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.431+0000 7ff6e5317700 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.113:6835/436972884,v1:172.21.15.113:6837/436972884] conn(0x557a76b94060 0x557a76b921b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a90e4aad2e5ad79b server_cookie=4ef83d5bdb292832 in_seq=0 out_seq=0 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.431+0000 7ff6bffff700 10 client.5178 ms_handle_connect on v2:172.21.15.113:6835/436972884 2024-01-15T11:26:17.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6bffff700 1 -- 172.21.15.113:0/2791285718 <== mds.0 v2:172.21.15.113:6835/436972884 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557a76adf960 con 0x557a76b94060 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6bffff700 10 client.5178 handle_command_reply: tid=0 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6f3777b80 1 client.5178 shutdown 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2791285718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 msgr2=0x557a76bd0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 0x557a76bd0870 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00f520 tx=0x7ff6dc058ca0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2791285718 shutdown_connections 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.113:6835/436972884,v1:172.21.15.113:6837/436972884] conn(0x557a76b94060 0x557a76b921b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76bbc410 0x557a76be9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6bffff700 0 client.5178 ms_handle_reset on v2:172.21.15.113:6835/436972884 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b1fce0 0x557a76bd36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2791285718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76b91dd0 0x557a76bd0870 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2791285718 >> 172.21.15.113:0/2791285718 conn(0x557a76ac1410 msgr2=0x557a76bbea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2791285718 shutdown_connections 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2791285718 wait complete. 2024-01-15T11:26:17.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.432+0000 7ff6f3777b80 20 client.5178 trim_cache size 0 max 16384 2024-01-15T11:26:17.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.450+0000 7ff6f3777b80 1 Processor -- start 2024-01-15T11:26:17.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.450+0000 7ff6f3777b80 1 -- start start 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6f3777b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b1fce0 0x557a76bb56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6f3777b80 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b91dd0 0x557a76bcda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6f3777b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 0x557a76bd1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6f3777b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a76965650 con 0x557a76bbc410 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6f3777b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a76b88e80 con 0x557a76b1fce0 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6f3777b80 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a76b88110 con 0x557a76b91dd0 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6e5b18700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 0x557a76bd1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6e5b18700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 0x557a76bd1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52082/0 (socket says 172.21.15.113:52082) 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6e5b18700 1 -- 172.21.15.113:0/4195549169 learned_addr learned my addr 172.21.15.113:0/4195549169 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6e5b18700 1 -- 172.21.15.113:0/4195549169 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b91dd0 msgr2=0x557a76bcda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/4195549169 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b91dd0 0x557a76bcda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6e5b18700 1 -- 172.21.15.113:0/4195549169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b1fce0 msgr2=0x557a76bb56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/4195549169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b1fce0 0x557a76bb56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6e5b18700 1 -- 172.21.15.113:0/4195549169 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a76967c80 con 0x557a76bbc410 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/4195549169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 0x557a76bd1180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8015040 tx=0x7ff6d800a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c13cf7f80e8fa47 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.451+0000 7ff6cb7fe700 1 -- 172.21.15.113:0/4195549169 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d800fa00 con 0x557a76bbc410 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.452+0000 7ff6cb7fe700 1 -- 172.21.15.113:0/4195549169 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d800fb60 con 0x557a76bbc410 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.452+0000 7ff6cb7fe700 1 -- 172.21.15.113:0/4195549169 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8001950 con 0x557a76bbc410 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.452+0000 7ff6f3777b80 1 -- 172.21.15.113:0/4195549169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 msgr2=0x557a76bd1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.452+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/4195549169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 0x557a76bd1180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8015040 tx=0x7ff6d800a000 comp rx=0 tx=0).stop 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.452+0000 7ff6f3777b80 1 -- 172.21.15.113:0/4195549169 shutdown_connections 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.452+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/4195549169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 0x557a76bd1180 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.452+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/4195549169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b1fce0 0x557a76bb56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.452+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/4195549169 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b91dd0 0x557a76bcda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.452+0000 7ff6f3777b80 1 -- 172.21.15.113:0/4195549169 >> 172.21.15.113:0/4195549169 conn(0x557a76ac1410 msgr2=0x557a76b8eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:17.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.452+0000 7ff6f3777b80 1 -- 172.21.15.113:0/4195549169 shutdown_connections 2024-01-15T11:26:17.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.452+0000 7ff6f3777b80 1 -- 172.21.15.113:0/4195549169 wait complete. 2024-01-15T11:26:17.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.453+0000 7ff6f3777b80 1 -- start start 2024-01-15T11:26:17.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.453+0000 7ff6f3777b80 1 Processor -- start 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.454+0000 7ff6f3777b80 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b1fce0 0x557a76c39230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.454+0000 7ff6f3777b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b91dd0 0x557a76c3cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.454+0000 7ff6f3777b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 0x557a76bb45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.454+0000 7ff6f3777b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a76bb5130 con 0x557a76bbc410 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.454+0000 7ff6f3777b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a76bc20f0 con 0x557a76b91dd0 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.454+0000 7ff6f3777b80 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a76c3c180 con 0x557a76b1fce0 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.454+0000 7ff6f3777b80 10 client.0 _ll_register_callbacks cb 0x7ff6dc059220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.454+0000 7ff6e5b18700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 0x557a76bb45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.454+0000 7ff6e5b18700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 0x557a76bb45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52088/0 (socket says 172.21.15.113:52088) 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.454+0000 7ff6e5b18700 1 -- 172.21.15.113:0/2873572908 learned_addr learned my addr 172.21.15.113:0/2873572908 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6e4b16700 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b91dd0 0x557a76c3cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6e5317700 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b1fce0 0x557a76c39230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6e5b18700 1 -- 172.21.15.113:0/2873572908 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b1fce0 msgr2=0x557a76c39230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b1fce0 0x557a76c39230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6e5b18700 1 -- 172.21.15.113:0/2873572908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b91dd0 msgr2=0x557a76c3cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b91dd0 0x557a76c3cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6e5b18700 1 -- 172.21.15.113:0/2873572908 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557a7690fb10 con 0x557a76bbc410 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6e5317700 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b1fce0 0x557a76c39230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6e4b16700 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b91dd0 0x557a76c3cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6e5b18700 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 0x557a76bb45d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00d1b00 tx=0x7ff6d8001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86581660e8ec7063 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6bffff700 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6bffff700 1 -- 172.21.15.113:0/2873572908 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d80057f0 con 0x557a76bbc410 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6bffff700 1 -- 172.21.15.113:0/2873572908 <== mon.0 v2:172.21.15.113:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6d8027470 con 0x557a76bbc410 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2873572908 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff6d800d050 con 0x557a76bbc410 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6bffff700 1 -- 172.21.15.113:0/2873572908 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6d8027630 con 0x557a76bbc410 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6f3777b80 10 client.5184 fetch_fsmap learned FSMap version 17 2024-01-15T11:26:17.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2873572908 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x557a7690dfe0 con 0x557a76bbc410 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6bffff700 1 -- 172.21.15.113:0/2873572908 <== mon.0 v2:172.21.15.113:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d800d050 con 0x557a76bbc410 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6bffff700 1 -- 172.21.15.113:0/2873572908 <== mon.0 v2:172.21.15.113:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff6d8012070 con 0x557a76bbc410 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6f3777b80 10 client.5184 fetch_fsmap finished waiting for FSMap version 17 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6f3777b80 10 client.5184 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6f3777b80 20 client.5184 populate_metadata read hostname 'smithi113' 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.113:6835/436972884,v1:172.21.15.113:6837/436972884] conn(0x557a76bd1570 0x557a76beb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6f3777b80 4 client.5184 mds_command: new command op to 5004 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2873572908 --> [v2:172.21.15.113:6835/436972884,v1:172.21.15.113:6837/436972884] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557a76adf680 con 0x557a76bd1570 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.455+0000 7ff6e5317700 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.113:6835/436972884,v1:172.21.15.113:6837/436972884] conn(0x557a76bd1570 0x557a76beb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6bffff700 10 client.5184 ms_handle_connect on v2:172.21.15.113:6835/436972884 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6e5317700 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.113:6835/436972884,v1:172.21.15.113:6837/436972884] conn(0x557a76bd1570 0x557a76beb340 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba98b662157baaeb server_cookie=1384ac3f887e9e87 in_seq=0 out_seq=0 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6bffff700 1 -- 172.21.15.113:0/2873572908 <== mds.0 v2:172.21.15.113:6835/436972884 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x557a76adf680 con 0x557a76bd1570 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6bffff700 10 client.5184 handle_command_reply: tid=0 2024-01-15T11:26:17.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6f3777b80 1 client.5184 shutdown 2024-01-15T11:26:17.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2873572908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 msgr2=0x557a76bb45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 0x557a76bb45d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00d1b00 tx=0x7ff6d8001510 comp rx=0 tx=0).stop 2024-01-15T11:26:17.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2873572908 shutdown_connections 2024-01-15T11:26:17.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a76bbc410 0x557a76bb45d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x557a76b1fce0 0x557a76c39230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.113:6835/436972884,v1:172.21.15.113:6837/436972884] conn(0x557a76bd1570 0x557a76beb340 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6f3777b80 1 --2- 172.21.15.113:0/2873572908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557a76b91dd0 0x557a76c3cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6bffff700 0 client.5184 ms_handle_reset on v2:172.21.15.113:6835/436972884 2024-01-15T11:26:17.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2873572908 >> 172.21.15.113:0/2873572908 conn(0x557a76ac1410 msgr2=0x557a76bc19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:17.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2873572908 shutdown_connections 2024-01-15T11:26:17.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6f3777b80 1 -- 172.21.15.113:0/2873572908 wait complete. 2024-01-15T11:26:17.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.456+0000 7ff6f3777b80 20 client.5184 trim_cache size 0 max 16384 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: "cluster_fsid": "910f8628-cd75-4a37-bf4e-e3603a95290d", 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: "whoami": 0, 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: "id": 5004, 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: "want_state": "up:active", 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "up:active", 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: "fs_name": "cephfs", 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: "rank_uptime": 5.9196790210000003, 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: "mdsmap_epoch": 17, 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_epoch": 33, 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_epoch_barrier": 33, 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: "uptime": 13.789629966 2024-01-15T11:26:17.462 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:26:17.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.457+0000 7ff6e757b700 1 -- 172.21.15.113:0/2002513662 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff6c406b350 msgr2=0x7ff6c406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.457+0000 7ff6e757b700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff6c406b350 0x7ff6c406d810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0015040 tx=0x7ff6d0007000 comp rx=0 tx=0).stop 2024-01-15T11:26:17.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.457+0000 7ff6e757b700 1 -- 172.21.15.113:0/2002513662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e015da50 msgr2=0x7ff6e0144530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.457+0000 7ff6e757b700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e015da50 0x7ff6e0144530 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0119c0 tx=0x7ff6dc011d80 comp rx=0 tx=0).stop 2024-01-15T11:26:17.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.457+0000 7ff6e757b700 1 -- 172.21.15.113:0/2002513662 shutdown_connections 2024-01-15T11:26:17.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.457+0000 7ff6e757b700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff6c406b350 0x7ff6c406d810 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.457+0000 7ff6e757b700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff6e00cef70 0x7ff6e00ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.457+0000 7ff6e757b700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6cc003a30 0x7ff6e00cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.457+0000 7ff6e757b700 1 --2- 172.21.15.113:0/2002513662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e015da50 0x7ff6e0144530 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.457+0000 7ff6e757b700 1 -- 172.21.15.113:0/2002513662 >> 172.21.15.113:0/2002513662 conn(0x7ff6e00b3130 msgr2=0x7ff6e00b0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:17.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.457+0000 7ff6e757b700 1 -- 172.21.15.113:0/2002513662 shutdown_connections 2024-01-15T11:26:17.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.457+0000 7ff6e757b700 1 -- 172.21.15.113:0/2002513662 wait complete. 2024-01-15T11:26:17.468 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T11:26:17.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.591+0000 7f25e0463700 1 Processor -- start 2024-01-15T11:26:17.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.591+0000 7f25e0463700 1 -- start start 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.592+0000 7f25e0463700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f25d80b19f0 0x7f25d81603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.592+0000 7f25e0463700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d80cd100 con 0x7f25d80d2160 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.592+0000 7f25e0463700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d80d09a0 con 0x7f25d815bed0 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.592+0000 7f25e0463700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d80b1e80 con 0x7f25d80b19f0 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.592+0000 7f25dd9fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f25d80d2160 0x7f25d80cd590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56510/0 (socket says 172.21.15.113:56510) 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.592+0000 7f25dd9fe700 1 -- 172.21.15.113:0/3722728425 learned_addr learned my addr 172.21.15.113:0/3722728425 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dea00700 1 --2- 172.21.15.113:0/3722728425 >> v2:172.21.15.141:3300/0 conn(0x7f25d80b19f0 0x7f25d81603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194310497 0 0) 0x7f25d80cd100 con 0x7f25d80d2160 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c40035b0 con 0x7f25d80d2160 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486618381 0 0) 0x7f25d80d09a0 con 0x7f25d815bed0 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25d80cd100 con 0x7f25d815bed0 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3347407235 0 0) 0x7f25c40035b0 con 0x7f25d80d2160 2024-01-15T11:26:17.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 >> v2:172.21.15.141:3300/0 conn(0x7f25d80b19f0 msgr2=0x7f25d81603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 --2- 172.21.15.113:0/3722728425 >> v2:172.21.15.141:3300/0 conn(0x7f25d80b19f0 0x7f25d81603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 >> v1:172.21.15.141:6789/0 conn(0x7f25d815bed0 legacy=0x7f25d80d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d80cc7d0 con 0x7f25d80d2160 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f25d4001670 con 0x7f25d80d2160 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dea00700 1 --2- 172.21.15.113:0/3722728425 >> v2:172.21.15.141:3300/0 conn(0x7f25d80b19f0 0x7f25d81603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 >> v1:172.21.15.113:6789/0 conn(0x7f25d80d2160 legacy=0x7f25d80cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 --2- 172.21.15.113:0/3722728425 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25c4003be0 0x7f25c4003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 --2- 172.21.15.113:0/3722728425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4004850 0x7f25c4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 --2- 172.21.15.113:0/3722728425 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25c4005210 0x7f25c400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d80b1e80 con 0x7f25c4004850 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c400c0e0 con 0x7f25c4003be0 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c400c2a0 con 0x7f25c4005210 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd9fe700 1 --2- 172.21.15.113:0/3722728425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4004850 0x7f25c4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25de1ff700 1 --2- 172.21.15.113:0/3722728425 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25c4003be0 0x7f25c4003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dea00700 1 --2- 172.21.15.113:0/3722728425 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25c4005210 0x7f25c400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd9fe700 1 -- 172.21.15.113:0/3722728425 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25c4005210 msgr2=0x7f25c400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd9fe700 1 --2- 172.21.15.113:0/3722728425 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25c4005210 0x7f25c400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd9fe700 1 -- 172.21.15.113:0/3722728425 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25c4003be0 msgr2=0x7f25c4003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd9fe700 1 --2- 172.21.15.113:0/3722728425 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25c4003be0 0x7f25c4003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dd9fe700 1 -- 172.21.15.113:0/3722728425 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c40035b0 con 0x7f25c4004850 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.593+0000 7f25dea00700 1 -- 172.21.15.113:0/3722728425 reap_dead start 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.594+0000 7f25dd9fe700 1 --2- 172.21.15.113:0/3722728425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4004850 0x7f25c4004cd0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f25d400c0e0 tx=0x7f25c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74df8294286f3ae8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.594+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d400cc40 con 0x7f25c4004850 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.594+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d400cda0 con 0x7f25c4004850 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.594+0000 7f25dd1fd700 1 -- 172.21.15.113:0/3722728425 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d4005ba0 con 0x7f25c4004850 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.594+0000 7f25e0463700 1 -- 172.21.15.113:0/3722728425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4004850 msgr2=0x7f25c4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.594+0000 7f25e0463700 1 --2- 172.21.15.113:0/3722728425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4004850 0x7f25c4004cd0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f25d400c0e0 tx=0x7f25c4005880 comp rx=0 tx=0).stop 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.595+0000 7f25e0463700 1 -- 172.21.15.113:0/3722728425 shutdown_connections 2024-01-15T11:26:17.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.595+0000 7f25e0463700 1 --2- 172.21.15.113:0/3722728425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4004850 0x7f25c4004cd0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.595+0000 7f25e0463700 1 -- 172.21.15.113:0/3722728425 >> 172.21.15.113:0/3722728425 conn(0x7f25d8073130 msgr2=0x7f25d809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:17.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.595+0000 7f25e0463700 1 -- 172.21.15.113:0/3722728425 shutdown_connections 2024-01-15T11:26:17.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.595+0000 7f25e0463700 1 -- 172.21.15.113:0/3722728425 wait complete. 2024-01-15T11:26:17.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.596+0000 7f25e0463700 1 Processor -- start 2024-01-15T11:26:17.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.596+0000 7f25e0463700 1 -- start start 2024-01-15T11:26:17.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.596+0000 7f25e0463700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25c4004850 0x7f25d81f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.596+0000 7f25e0463700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d8158480 0x7f25d80b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.596+0000 7f25e0463700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25d814c390 0x7f25d814c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.596+0000 7f25e0463700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d815c5b0 con 0x7f25d8158480 2024-01-15T11:26:17.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.596+0000 7f25e0463700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d80b7160 con 0x7f25c4004850 2024-01-15T11:26:17.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.596+0000 7f25e0463700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d815f0e0 con 0x7f25d814c390 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25dd9fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d8158480 0x7f25d80b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25dd9fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d8158480 0x7f25d80b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52106/0 (socket says 172.21.15.113:52106) 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25dd9fe700 1 -- 172.21.15.113:0/1990821354 learned_addr learned my addr 172.21.15.113:0/1990821354 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25de1ff700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25c4004850 0x7f25d81f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25dea00700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25d814c390 0x7f25d814c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25dd9fe700 1 -- 172.21.15.113:0/1990821354 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25d814c390 msgr2=0x7f25d814c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25dd9fe700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25d814c390 0x7f25d814c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25dd9fe700 1 -- 172.21.15.113:0/1990821354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25c4004850 msgr2=0x7f25d81f6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25dd9fe700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25c4004850 0x7f25d81f6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25dd9fe700 1 -- 172.21.15.113:0/1990821354 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d815eb20 con 0x7f25d8158480 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25dea00700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25d814c390 0x7f25d814c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25de1ff700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25c4004850 0x7f25d81f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25dd9fe700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d8158480 0x7f25d80b4870 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f25c40047d0 tx=0x7f25d4001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25ceffd700 1 -- 172.21.15.113:0/1990821354 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d40072d0 con 0x7f25d8158480 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25ceffd700 1 -- 172.21.15.113:0/1990821354 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d4007450 con 0x7f25d8158480 2024-01-15T11:26:17.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25e0463700 1 -- 172.21.15.113:0/1990821354 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d815ee00 con 0x7f25d8158480 2024-01-15T11:26:17.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25ceffd700 1 -- 172.21.15.113:0/1990821354 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d4007800 con 0x7f25d8158480 2024-01-15T11:26:17.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.597+0000 7f25e0463700 1 -- 172.21.15.113:0/1990821354 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d80b75f0 con 0x7f25d8158480 2024-01-15T11:26:17.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.598+0000 7f25e0463700 1 -- 172.21.15.113:0/1990821354 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d80b72d0 con 0x7f25d8158480 2024-01-15T11:26:17.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.599+0000 7f25ceffd700 1 -- 172.21.15.113:0/1990821354 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f25d401e430 con 0x7f25d8158480 2024-01-15T11:26:17.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.599+0000 7f25ceffd700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f25bc06b350 0x7f25bc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:17.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.599+0000 7f25ceffd700 1 -- 172.21.15.113:0/1990821354 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25d408ac70 con 0x7f25d8158480 2024-01-15T11:26:17.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.599+0000 7f25de1ff700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f25bc06b350 0x7f25bc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:17.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.599+0000 7f25de1ff700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f25bc06b350 0x7f25bc06d810 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f25c8007fd0 tx=0x7f25c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:17.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.603+0000 7f25ceffd700 1 -- 172.21.15.113:0/1990821354 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25d4055b70 con 0x7f25d8158480 2024-01-15T11:26:17.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.844+0000 7f25e0463700 1 -- 172.21.15.113:0/1990821354 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f25d8067f60 con 0x7f25d8158480 2024-01-15T11:26:17.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.845+0000 7f25ceffd700 1 -- 172.21.15.113:0/1990821354 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f25d4017070 con 0x7f25d8158480 2024-01-15T11:26:17.844 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T11:26:17.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.847+0000 7f25e0463700 1 -- 172.21.15.113:0/1990821354 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f25bc06b350 msgr2=0x7f25bc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.847+0000 7f25e0463700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f25bc06b350 0x7f25bc06d810 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f25c8007fd0 tx=0x7f25c8009040 comp rx=0 tx=0).stop 2024-01-15T11:26:17.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.847+0000 7f25e0463700 1 -- 172.21.15.113:0/1990821354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d8158480 msgr2=0x7f25d80b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:17.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.847+0000 7f25e0463700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d8158480 0x7f25d80b4870 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f25c40047d0 tx=0x7f25d4001930 comp rx=0 tx=0).stop 2024-01-15T11:26:17.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.847+0000 7f25e0463700 1 -- 172.21.15.113:0/1990821354 shutdown_connections 2024-01-15T11:26:17.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.847+0000 7f25e0463700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f25bc06b350 0x7f25bc06d810 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.847+0000 7f25e0463700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25d814c390 0x7f25d814c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.847+0000 7f25e0463700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25c4004850 0x7f25d81f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.847+0000 7f25e0463700 1 --2- 172.21.15.113:0/1990821354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d8158480 0x7f25d80b4870 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:17.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.847+0000 7f25e0463700 1 -- 172.21.15.113:0/1990821354 >> 172.21.15.113:0/1990821354 conn(0x7f25d8073130 msgr2=0x7f25d80cede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:17.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.847+0000 7f25e0463700 1 -- 172.21.15.113:0/1990821354 shutdown_connections 2024-01-15T11:26:17.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:17.847+0000 7f25e0463700 1 -- 172.21.15.113:0/1990821354 wait complete. 2024-01-15T11:26:17.858 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-15T11:26:17.858 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T11:26:17.858 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi113.front.sepia.ceph.com 2024-01-15T11:26:17.858 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T11:26:17.858 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T11:26:17.859 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-15T11:26:17.859 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-15T11:26:17.859 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T11:26:17.859 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T11:26:17.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:17.859 DEBUG:teuthology.orchestra.run.smithi113:> ip addr 2024-01-15T11:26:17.876 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T11:26:17.876 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T11:26:17.876 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T11:26:17.876 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:17.876 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-01-15T11:26:17.876 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:17.876 INFO:teuthology.orchestra.run.smithi113.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:26:17.876 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:44 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f0 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:45 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f1 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c0 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c1 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: inet 172.21.15.113/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft 42447sec preferred_lft 42447sec 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::105b:ecb2:15ad:83c/64 scope link noprefixroute 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 26:9b:2b:80:b1:f2 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:17.877 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::b017:17ff:fe98:c0a9/64 scope link 2024-01-15T11:26:17.878 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:17.878 INFO:teuthology.orchestra.run.smithi113.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T11:26:17.878 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 26:9b:2b:80:b1:f2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T11:26:17.878 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::249b:2bff:fe80:b1f2/64 scope link 2024-01-15T11:26:17.878 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:17.878 INFO:teuthology.orchestra.run.smithi113.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T11:26:17.878 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 9a:d8:5d:36:e7:20 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T11:26:17.878 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::98d8:5dff:fe36:e720/64 scope link 2024-01-15T11:26:17.878 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:17.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:17.878 DEBUG:teuthology.orchestra.run.smithi113:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T11:26:17.941 INFO:teuthology.orchestra.run.smithi113.stdout:1 2024-01-15T11:26:17.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:17.943 DEBUG:teuthology.orchestra.run.smithi113:> route 2024-01-15T11:26:18.001 INFO:teuthology.orchestra.run.smithi113.stdout:Kernel IP routing table 2024-01-15T11:26:18.002 INFO:teuthology.orchestra.run.smithi113.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T11:26:18.002 INFO:teuthology.orchestra.run.smithi113.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T11:26:18.002 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T11:26:18.002 INFO:teuthology.orchestra.run.smithi113.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T11:26:18.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:18.002 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-15T11:26:18.002 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-15T11:26:18.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T11:26:18.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T11:26:18.003 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T11:26:18.003 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-15T11:26:18.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:18.107 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-01-15T11:26:18.124 INFO:teuthology.orchestra.run.smithi113.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T11:26:18.124 INFO:teuthology.orchestra.run.smithi113.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T11:26:18.124 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T11:26:18.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:18.124 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set brx.0 up 2024-01-15T11:26:18.191 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T11:26:18.191 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T11:26:18.249 INFO:teuthology.orchestra.run.smithi113.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T11:26:18.249 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T11:26:18.249 DEBUG:teuthology.orchestra.run.smithi113:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T11:26:18.305 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-15T11:26:18.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:18.305 DEBUG:teuthology.orchestra.run.smithi113:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-15T11:26:18.391 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-15T11:26:18.392 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-15T11:26:18.392 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-15T11:26:18.401 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-15T11:26:18.401 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-15T11:26:18.401 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-15T11:26:18.401 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.141:3300,172.21.15.141:3301" 2024-01-15T11:26:18.401 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T11:26:18.419 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-15T11:26:18.419 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-15T11:26:18.419 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-15T11:26:18.419 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.141:3300,172.21.15.141:3301" 2024-01-15T11:26:18.419 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T11:26:18.419 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:6789,172.21.15.141:6789,172.21.15.141:6790" 2024-01-15T11:26:18.419 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.113:6789,172.21.15.141:6789,172.21.15.141:6790:/ 2024-01-15T11:26:18.419 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=910f8628-cd75-4a37-bf4e-e3603a95290d" will pass to kernel 2024-01-15T11:26:18.421 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-15T11:26:18.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:18.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T11:26:18.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:18.448 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:26:18.535 INFO:teuthology.orchestra.run.smithi113.stdout:client5050 2024-01-15T11:26:18.535 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:26:18.536 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:26:18.553 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T11:26:18.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.716+0000 7fe4d6852700 1 Processor -- start 2024-01-15T11:26:18.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.716+0000 7fe4d6852700 1 -- start start 2024-01-15T11:26:18.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4d6852700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe4d015e080 0x7fe4d00cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:18.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4d6852700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe4d00d22f0 0x7fe4d00d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:18.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4d6852700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d00d0150 con 0x7fe4d00ce4d0 2024-01-15T11:26:18.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4d6852700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d015e460 con 0x7fe4d015e080 2024-01-15T11:26:18.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4d6852700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d00cee50 con 0x7fe4d00d22f0 2024-01-15T11:26:18.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4cffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe4d015e080 0x7fe4d00cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:18.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4cffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe4d015e080 0x7fe4d00cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52122/0 (socket says 172.21.15.113:52122) 2024-01-15T11:26:18.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4cffff700 1 -- 172.21.15.113:0/772884362 learned_addr learned my addr 172.21.15.113:0/772884362 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:18.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4cffff700 1 -- 172.21.15.113:0/772884362 >> v2:172.21.15.141:3300/0 conn(0x7fe4d00d22f0 msgr2=0x7fe4d00d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:18.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4cffff700 1 --2- 172.21.15.113:0/772884362 >> v2:172.21.15.141:3300/0 conn(0x7fe4d00d22f0 0x7fe4d00d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:18.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4cffff700 1 -- 172.21.15.113:0/772884362 >> v1:172.21.15.141:6789/0 conn(0x7fe4d00ce4d0 legacy=0x7fe4d00d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:18.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4cffff700 1 -- 172.21.15.113:0/772884362 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d00cf130 con 0x7fe4d015e080 2024-01-15T11:26:18.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4cffff700 1 --2- 172.21.15.113:0/772884362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d015e080 0x7fe4d00cc2a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00c8cb0 tx=0x7fe4c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b160c32a49d9ecec server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:18.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4ceffd700 1 -- 172.21.15.113:0/772884362 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0004230 con 0x7fe4d015e080 2024-01-15T11:26:18.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4ceffd700 1 -- 172.21.15.113:0/772884362 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c0004390 con 0x7fe4d015e080 2024-01-15T11:26:18.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.717+0000 7fe4ceffd700 1 -- 172.21.15.113:0/772884362 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c00036a0 con 0x7fe4d015e080 2024-01-15T11:26:18.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.718+0000 7fe4d6852700 1 -- 172.21.15.113:0/772884362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d015e080 msgr2=0x7fe4d00cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:18.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.718+0000 7fe4d6852700 1 --2- 172.21.15.113:0/772884362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d015e080 0x7fe4d00cc2a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00c8cb0 tx=0x7fe4c0009bd0 comp rx=0 tx=0).stop 2024-01-15T11:26:18.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.718+0000 7fe4d6852700 1 -- 172.21.15.113:0/772884362 shutdown_connections 2024-01-15T11:26:18.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.718+0000 7fe4d6852700 1 --2- 172.21.15.113:0/772884362 >> v2:172.21.15.141:3300/0 conn(0x7fe4d00d22f0 0x7fe4d00d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:18.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.718+0000 7fe4d6852700 1 --2- 172.21.15.113:0/772884362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d015e080 0x7fe4d00cc2a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:18.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.718+0000 7fe4d6852700 1 -- 172.21.15.113:0/772884362 >> 172.21.15.113:0/772884362 conn(0x7fe4d0073130 msgr2=0x7fe4d009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:18.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.718+0000 7fe4d6852700 1 -- 172.21.15.113:0/772884362 shutdown_connections 2024-01-15T11:26:18.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.718+0000 7fe4d6852700 1 -- 172.21.15.113:0/772884362 wait complete. 2024-01-15T11:26:18.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.719+0000 7fe4d6852700 1 Processor -- start 2024-01-15T11:26:18.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.720+0000 7fe4d6852700 1 -- start start 2024-01-15T11:26:18.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d6852700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4d00ce4d0 0x7fe4d00b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d6852700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe4d00d22f0 0x7fe4d00c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d6852700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d015e080 0x7fe4d00cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d6852700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00b60f0 con 0x7fe4d015e080 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d6852700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00b5470 con 0x7fe4d00ce4d0 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d6852700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00b5750 con 0x7fe4d00d22f0 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d4def700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d015e080 0x7fe4d00cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d4def700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d015e080 0x7fe4d00cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52132/0 (socket says 172.21.15.113:52132) 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d4def700 1 -- 172.21.15.113:0/1219909873 learned_addr learned my addr 172.21.15.113:0/1219909873 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4cffff700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4d00ce4d0 0x7fe4d00b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4cf7fe700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe4d00d22f0 0x7fe4d00c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d4def700 1 -- 172.21.15.113:0/1219909873 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe4d00d22f0 msgr2=0x7fe4d00c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d4def700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe4d00d22f0 0x7fe4d00c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:18.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d4def700 1 -- 172.21.15.113:0/1219909873 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4d00ce4d0 msgr2=0x7fe4d00b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d4def700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4d00ce4d0 0x7fe4d00b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d4def700 1 -- 172.21.15.113:0/1219909873 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c0009550 con 0x7fe4d015e080 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4cf7fe700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe4d00d22f0 0x7fe4d00c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d4def700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d015e080 0x7fe4d00cacd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800b970 tx=0x7fe4c800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4ccff9700 1 -- 172.21.15.113:0/1219909873 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c800c930 con 0x7fe4d015e080 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4ccff9700 1 -- 172.21.15.113:0/1219909873 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c800cab0 con 0x7fe4d015e080 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4ccff9700 1 -- 172.21.15.113:0/1219909873 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c800ce60 con 0x7fe4d015e080 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d6852700 1 -- 172.21.15.113:0/1219909873 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d00b5a30 con 0x7fe4d015e080 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.721+0000 7fe4d6852700 1 -- 172.21.15.113:0/1219909873 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d00c94d0 con 0x7fe4d015e080 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.722+0000 7fe4d6852700 1 -- 172.21.15.113:0/1219909873 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d00b6230 con 0x7fe4d015e080 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.723+0000 7fe4ccff9700 1 -- 172.21.15.113:0/1219909873 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe4c801d3b0 con 0x7fe4d015e080 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.723+0000 7fe4ccff9700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe4bc06b350 0x7fe4bc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:18.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.723+0000 7fe4ccff9700 1 -- 172.21.15.113:0/1219909873 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4c8006e40 con 0x7fe4d015e080 2024-01-15T11:26:18.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.723+0000 7fe4cffff700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe4bc06b350 0x7fe4bc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:18.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.724+0000 7fe4cffff700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe4bc06b350 0x7fe4bc06d810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0009130 tx=0x7fe4c0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:18.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.728+0000 7fe4ccff9700 1 -- 172.21.15.113:0/1219909873 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4d00b6230 con 0x7fe4d015e080 2024-01-15T11:26:18.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.963+0000 7fe4d6852700 1 -- 172.21.15.113:0/1219909873 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe4d00b6230 con 0x7fe4d015e080 2024-01-15T11:26:18.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.963+0000 7fe4ccff9700 1 -- 172.21.15.113:0/1219909873 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe4d00b6230 con 0x7fe4d015e080 2024-01-15T11:26:18.962 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T11:26:18.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.965+0000 7fe4d6852700 1 -- 172.21.15.113:0/1219909873 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe4bc06b350 msgr2=0x7fe4bc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.965+0000 7fe4d6852700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe4bc06b350 0x7fe4bc06d810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0009130 tx=0x7fe4c0008030 comp rx=0 tx=0).stop 2024-01-15T11:26:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.965+0000 7fe4d6852700 1 -- 172.21.15.113:0/1219909873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d015e080 msgr2=0x7fe4d00cacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.965+0000 7fe4d6852700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d015e080 0x7fe4d00cacd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800b970 tx=0x7fe4c800bc80 comp rx=0 tx=0).stop 2024-01-15T11:26:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.965+0000 7fe4d6852700 1 -- 172.21.15.113:0/1219909873 shutdown_connections 2024-01-15T11:26:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.965+0000 7fe4d6852700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe4bc06b350 0x7fe4bc06d810 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.965+0000 7fe4d6852700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d015e080 0x7fe4d00cacd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.965+0000 7fe4d6852700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe4d00ce4d0 0x7fe4d00b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.965+0000 7fe4d6852700 1 --2- 172.21.15.113:0/1219909873 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe4d00d22f0 0x7fe4d00c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.965+0000 7fe4d6852700 1 -- 172.21.15.113:0/1219909873 >> 172.21.15.113:0/1219909873 conn(0x7fe4d0073130 msgr2=0x7fe4d009ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.965+0000 7fe4d6852700 1 -- 172.21.15.113:0/1219909873 shutdown_connections 2024-01-15T11:26:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:18.965+0000 7fe4d6852700 1 -- 172.21.15.113:0/1219909873 wait complete. 2024-01-15T11:26:18.976 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-15T11:26:18.976 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T11:26:18.977 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi141.front.sepia.ceph.com 2024-01-15T11:26:18.977 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T11:26:18.977 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T11:26:18.977 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-15T11:26:18.977 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-15T11:26:18.977 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T11:26:18.977 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T11:26:18.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:18.977 DEBUG:teuthology.orchestra.run.smithi141:> ip addr 2024-01-15T11:26:18.999 INFO:teuthology.orchestra.run.smithi141.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T11:26:18.999 INFO:teuthology.orchestra.run.smithi141.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T11:26:18.999 INFO:teuthology.orchestra.run.smithi141.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T11:26:18.999 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:18.999 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 ::1/128 scope host 2024-01-15T11:26:18.999 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:18.999 INFO:teuthology.orchestra.run.smithi141.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:26:18.999 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:26 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:26:18.999 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f0 2024-01-15T11:26:18.999 INFO:teuthology.orchestra.run.smithi141.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:26:18.999 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:d9:3b:27 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: altname enp7s0f1 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6c brd ff:ff:ff:ff:ff:ff 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 0c:c4:7a:88:72:6d brd ff:ff:ff:ff:ff:ff 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: inet 172.21.15.141/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft 42415sec preferred_lft 42415sec 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::91fc:8187:19e2:bf82/64 scope link noprefixroute 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 1a:e8:62:f1:ab:25 brd ff:ff:ff:ff:ff:ff 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::cc20:9bff:fea4:a969/64 scope link 2024-01-15T11:26:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:19.001 INFO:teuthology.orchestra.run.smithi141.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T11:26:19.001 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether 1a:e8:62:f1:ab:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T11:26:19.001 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::18e8:62ff:fef1:ab25/64 scope link 2024-01-15T11:26:19.001 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:19.001 INFO:teuthology.orchestra.run.smithi141.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T11:26:19.001 INFO:teuthology.orchestra.run.smithi141.stdout: link/ether ca:25:68:32:50:e2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T11:26:19.001 INFO:teuthology.orchestra.run.smithi141.stdout: inet6 fe80::c825:68ff:fe32:50e2/64 scope link 2024-01-15T11:26:19.001 INFO:teuthology.orchestra.run.smithi141.stdout: valid_lft forever preferred_lft forever 2024-01-15T11:26:19.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:19.001 DEBUG:teuthology.orchestra.run.smithi141:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T11:26:19.069 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-15T11:26:19.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:19.071 DEBUG:teuthology.orchestra.run.smithi141:> route 2024-01-15T11:26:19.133 INFO:teuthology.orchestra.run.smithi141.stdout:Kernel IP routing table 2024-01-15T11:26:19.134 INFO:teuthology.orchestra.run.smithi141.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T11:26:19.134 INFO:teuthology.orchestra.run.smithi141.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T11:26:19.134 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T11:26:19.134 INFO:teuthology.orchestra.run.smithi141.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T11:26:19.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:19.134 DEBUG:teuthology.orchestra.run.smithi141:> (cd / && exec bash -c ' 2024-01-15T11:26:19.134 DEBUG:teuthology.orchestra.run.smithi141:> set -e 2024-01-15T11:26:19.134 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T11:26:19.134 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T11:26:19.134 DEBUG:teuthology.orchestra.run.smithi141:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T11:26:19.134 DEBUG:teuthology.orchestra.run.smithi141:> ') 2024-01-15T11:26:19.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:19.247 DEBUG:teuthology.orchestra.run.smithi141:> ip netns list 2024-01-15T11:26:19.265 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T11:26:19.265 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T11:26:19.265 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T11:26:19.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:19.265 DEBUG:teuthology.orchestra.run.smithi141:> sudo ip link set brx.0 up 2024-01-15T11:26:19.333 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T11:26:19.333 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T11:26:19.390 INFO:teuthology.orchestra.run.smithi141.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T11:26:19.390 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T11:26:19.390 DEBUG:teuthology.orchestra.run.smithi141:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T11:26:19.446 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-15T11:26:19.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:19.447 DEBUG:teuthology.orchestra.run.smithi141:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-15T11:26:19.533 INFO:teuthology.orchestra.run.smithi141.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-15T11:26:19.533 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-15T11:26:19.533 INFO:teuthology.orchestra.run.smithi141.stdout:invalid new device string format 2024-01-15T11:26:19.542 INFO:teuthology.orchestra.run.smithi141.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-15T11:26:19.542 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-15T11:26:19.542 INFO:teuthology.orchestra.run.smithi141.stdout:invalid new device string format 2024-01-15T11:26:19.542 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.141:3300,172.21.15.141:3301" 2024-01-15T11:26:19.543 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T11:26:19.561 INFO:teuthology.orchestra.run.smithi141.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-15T11:26:19.561 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-15T11:26:19.561 INFO:teuthology.orchestra.run.smithi141.stdout:invalid new device string format 2024-01-15T11:26:19.561 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.141:3300,172.21.15.141:3301" 2024-01-15T11:26:19.561 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T11:26:19.561 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: resolved to: "172.21.15.113:6789,172.21.15.141:6789,172.21.15.141:6790" 2024-01-15T11:26:19.561 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.113:6789,172.21.15.141:6789,172.21.15.141:6790:/ 2024-01-15T11:26:19.561 INFO:teuthology.orchestra.run.smithi141.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=910f8628-cd75-4a37-bf4e-e3603a95290d" will pass to kernel 2024-01-15T11:26:19.564 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-15T11:26:19.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:19.564 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T11:26:19.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:19.591 DEBUG:teuthology.orchestra.run.smithi141:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T11:26:19.665 INFO:teuthology.orchestra.run.smithi141.stdout:client5056 2024-01-15T11:26:19.665 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-15T11:26:19.665 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:26:19.724 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T11:26:19.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.889+0000 7fc6a53a5700 1 Processor -- start 2024-01-15T11:26:19.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.889+0000 7fc6a53a5700 1 -- start start 2024-01-15T11:26:19.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.889+0000 7fc6a53a5700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc6a015e150 0x7fc6a00b9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:19.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.889+0000 7fc6a53a5700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc6a0161bb0 0x7fc6a00ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:19.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc6a53a5700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6a00cf050 0x7fc6a00b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:19.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc6a53a5700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a015e530 con 0x7fc6a015e150 2024-01-15T11:26:19.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc6a53a5700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a00d21a0 con 0x7fc6a0161bb0 2024-01-15T11:26:19.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc6a53a5700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a00cf480 con 0x7fc6a00cf050 2024-01-15T11:26:19.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69effd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc6a015e150 0x7fc6a00b9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:19.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69effd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc6a015e150 0x7fc6a00b9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52142/0 (socket says 172.21.15.113:52142) 2024-01-15T11:26:19.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69effd700 1 -- 172.21.15.113:0/4230298262 learned_addr learned my addr 172.21.15.113:0/4230298262 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:19.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69f7fe700 1 --2- 172.21.15.113:0/4230298262 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6a00cf050 0x7fc6a00b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:19.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69e7fc700 1 --2- 172.21.15.113:0/4230298262 >> v2:172.21.15.141:3300/0 conn(0x7fc6a0161bb0 0x7fc6a00ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:19.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69effd700 1 -- 172.21.15.113:0/4230298262 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6a00cf050 msgr2=0x7fc6a00b58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:19.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69effd700 1 --2- 172.21.15.113:0/4230298262 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6a00cf050 0x7fc6a00b58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:19.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69effd700 1 -- 172.21.15.113:0/4230298262 >> v2:172.21.15.141:3300/0 conn(0x7fc6a0161bb0 msgr2=0x7fc6a00ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:19.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69effd700 1 --2- 172.21.15.113:0/4230298262 >> v2:172.21.15.141:3300/0 conn(0x7fc6a0161bb0 0x7fc6a00ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:19.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69effd700 1 -- 172.21.15.113:0/4230298262 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a0162390 con 0x7fc6a015e150 2024-01-15T11:26:19.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69f7fe700 1 --2- 172.21.15.113:0/4230298262 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6a00cf050 0x7fc6a00b58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:19.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69e7fc700 1 --2- 172.21.15.113:0/4230298262 >> v2:172.21.15.141:3300/0 conn(0x7fc6a0161bb0 0x7fc6a00ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:19.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69effd700 1 --2- 172.21.15.113:0/4230298262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6a015e150 0x7fc6a00b9a40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0072130 tx=0x7fc690009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec856bb9a4b4d4de server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:19.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69dffb700 1 -- 172.21.15.113:0/4230298262 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69000b870 con 0x7fc6a015e150 2024-01-15T11:26:19.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69dffb700 1 -- 172.21.15.113:0/4230298262 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69000b9d0 con 0x7fc6a015e150 2024-01-15T11:26:19.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.890+0000 7fc69dffb700 1 -- 172.21.15.113:0/4230298262 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69000bd60 con 0x7fc6a015e150 2024-01-15T11:26:19.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.891+0000 7fc6a53a5700 1 -- 172.21.15.113:0/4230298262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6a015e150 msgr2=0x7fc6a00b9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:19.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.891+0000 7fc6a53a5700 1 --2- 172.21.15.113:0/4230298262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6a015e150 0x7fc6a00b9a40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0072130 tx=0x7fc690009bd0 comp rx=0 tx=0).stop 2024-01-15T11:26:19.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.891+0000 7fc6a53a5700 1 -- 172.21.15.113:0/4230298262 shutdown_connections 2024-01-15T11:26:19.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.891+0000 7fc6a53a5700 1 --2- 172.21.15.113:0/4230298262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6a015e150 0x7fc6a00b9a40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:19.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.891+0000 7fc6a53a5700 1 --2- 172.21.15.113:0/4230298262 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6a00cf050 0x7fc6a00b58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:19.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.891+0000 7fc6a53a5700 1 --2- 172.21.15.113:0/4230298262 >> v2:172.21.15.141:3300/0 conn(0x7fc6a0161bb0 0x7fc6a00ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:19.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.891+0000 7fc6a53a5700 1 -- 172.21.15.113:0/4230298262 >> 172.21.15.113:0/4230298262 conn(0x7fc6a006b620 msgr2=0x7fc6a0071610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:19.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.891+0000 7fc6a53a5700 1 -- 172.21.15.113:0/4230298262 shutdown_connections 2024-01-15T11:26:19.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.891+0000 7fc6a53a5700 1 -- 172.21.15.113:0/4230298262 wait complete. 2024-01-15T11:26:19.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.893+0000 7fc6a53a5700 1 Processor -- start 2024-01-15T11:26:19.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.893+0000 7fc6a53a5700 1 -- start start 2024-01-15T11:26:19.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc6a53a5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6a00cf050 0x7fc6a00cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:19.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc6a53a5700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6a015e150 0x7fc6a015e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:19.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc6a53a5700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6a0161bb0 0x7fc6a014c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:19.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc6a53a5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a015f910 con 0x7fc6a00cf050 2024-01-15T11:26:19.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc6a53a5700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a015ed90 con 0x7fc6a015e150 2024-01-15T11:26:19.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc6a53a5700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a015f070 con 0x7fc6a0161bb0 2024-01-15T11:26:19.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc69effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6a00cf050 0x7fc6a00cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:19.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc69effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6a00cf050 0x7fc6a00cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52152/0 (socket says 172.21.15.113:52152) 2024-01-15T11:26:19.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc69effd700 1 -- 172.21.15.113:0/1320275567 learned_addr learned my addr 172.21.15.113:0/1320275567 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:19.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc69e7fc700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6a015e150 0x7fc6a015e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:19.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc69f7fe700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6a0161bb0 0x7fc6a014c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:19.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc69effd700 1 -- 172.21.15.113:0/1320275567 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6a0161bb0 msgr2=0x7fc6a014c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:19.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc69effd700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6a0161bb0 0x7fc6a014c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:19.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc69effd700 1 -- 172.21.15.113:0/1320275567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6a015e150 msgr2=0x7fc6a015e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:19.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc69effd700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6a015e150 0x7fc6a015e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:19.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc69effd700 1 -- 172.21.15.113:0/1320275567 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc690009550 con 0x7fc6a00cf050 2024-01-15T11:26:19.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc69effd700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6a00cf050 0x7fc6a00cdaa0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc690014040 tx=0x7fc690004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:19.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc6877fe700 1 -- 172.21.15.113:0/1320275567 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690004250 con 0x7fc6a00cf050 2024-01-15T11:26:19.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc6877fe700 1 -- 172.21.15.113:0/1320275567 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69001e070 con 0x7fc6a00cf050 2024-01-15T11:26:19.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc6a53a5700 1 -- 172.21.15.113:0/1320275567 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a015f350 con 0x7fc6a00cf050 2024-01-15T11:26:19.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc6877fe700 1 -- 172.21.15.113:0/1320275567 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6900036a0 con 0x7fc6a00cf050 2024-01-15T11:26:19.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.894+0000 7fc6a53a5700 1 -- 172.21.15.113:0/1320275567 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a014c5e0 con 0x7fc6a00cf050 2024-01-15T11:26:19.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.895+0000 7fc6a53a5700 1 -- 172.21.15.113:0/1320275567 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a014c770 con 0x7fc6a00cf050 2024-01-15T11:26:19.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.896+0000 7fc6877fe700 1 -- 172.21.15.113:0/1320275567 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc690020030 con 0x7fc6a00cf050 2024-01-15T11:26:19.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.896+0000 7fc6877fe700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc68c06b350 0x7fc68c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:19.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.896+0000 7fc6877fe700 1 -- 172.21.15.113:0/1320275567 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc69008abb0 con 0x7fc6a00cf050 2024-01-15T11:26:19.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.896+0000 7fc69e7fc700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc68c06b350 0x7fc68c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:19.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.897+0000 7fc69e7fc700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc68c06b350 0x7fc68c06d810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc6a015db30 tx=0x7fc694008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:19.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:19.901+0000 7fc6877fe700 1 -- 172.21.15.113:0/1320275567 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc690055ab0 con 0x7fc6a00cf050 2024-01-15T11:26:20.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.135+0000 7fc6a53a5700 1 -- 172.21.15.113:0/1320275567 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc6a00751e0 con 0x7fc6a00cf050 2024-01-15T11:26:20.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.594+0000 7fc6877fe700 1 -- 172.21.15.113:0/1320275567 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 136+0+0 (secure 0 0 0) 0x7fc6900590d0 con 0x7fc6a00cf050 2024-01-15T11:26:20.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.597+0000 7fc6a53a5700 1 -- 172.21.15.113:0/1320275567 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc68c06b350 msgr2=0x7fc68c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:20.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.597+0000 7fc6a53a5700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc68c06b350 0x7fc68c06d810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc6a015db30 tx=0x7fc694008040 comp rx=0 tx=0).stop 2024-01-15T11:26:20.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.597+0000 7fc6a53a5700 1 -- 172.21.15.113:0/1320275567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6a00cf050 msgr2=0x7fc6a00cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:20.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.597+0000 7fc6a53a5700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6a00cf050 0x7fc6a00cdaa0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc690014040 tx=0x7fc690004750 comp rx=0 tx=0).stop 2024-01-15T11:26:20.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.597+0000 7fc6a53a5700 1 -- 172.21.15.113:0/1320275567 shutdown_connections 2024-01-15T11:26:20.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.597+0000 7fc6a53a5700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc68c06b350 0x7fc68c06d810 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.597+0000 7fc6a53a5700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc6a0161bb0 0x7fc6a014c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.597+0000 7fc6a53a5700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6a00cf050 0x7fc6a00cdaa0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.597+0000 7fc6a53a5700 1 --2- 172.21.15.113:0/1320275567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc6a015e150 0x7fc6a015e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.597+0000 7fc6a53a5700 1 -- 172.21.15.113:0/1320275567 >> 172.21.15.113:0/1320275567 conn(0x7fc6a006b620 msgr2=0x7fc6a00d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:20.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.597+0000 7fc6a53a5700 1 -- 172.21.15.113:0/1320275567 shutdown_connections 2024-01-15T11:26:20.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.597+0000 7fc6a53a5700 1 -- 172.21.15.113:0/1320275567 wait complete. 2024-01-15T11:26:20.611 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T11:26:20.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.735+0000 7f7eefeee700 1 Processor -- start 2024-01-15T11:26:20.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.736+0000 7f7eefeee700 1 -- start start 2024-01-15T11:26:20.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.736+0000 7f7eefeee700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f7ee815c140 0x7f7ee80726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:20.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.736+0000 7f7eefeee700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee80adfd0 con 0x7f7ee80622a0 2024-01-15T11:26:20.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.736+0000 7f7eefeee700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee80ae2b0 con 0x7f7ee80af780 2024-01-15T11:26:20.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.736+0000 7f7eefeee700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee80afbf0 con 0x7f7ee815c140 2024-01-15T11:26:20.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.736+0000 7f7eed489700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f7ee80622a0 0x7f7ee80af070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56512/0 (socket says 172.21.15.113:56512) 2024-01-15T11:26:20.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eed489700 1 -- 172.21.15.113:0/4049527885 learned_addr learned my addr 172.21.15.113:0/4049527885 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:20.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825927730 0 0) 0x7f7ee80adfd0 con 0x7f7ee80622a0 2024-01-15T11:26:20.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed40035b0 con 0x7f7ee80622a0 2024-01-15T11:26:20.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eedc8a700 1 --2- 172.21.15.113:0/4049527885 >> v2:172.21.15.141:3300/0 conn(0x7f7ee815c140 0x7f7ee80726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:20.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4116536922 0 0) 0x7f7ed40035b0 con 0x7f7ee80622a0 2024-01-15T11:26:20.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 >> v2:172.21.15.141:3300/0 conn(0x7f7ee815c140 msgr2=0x7f7ee80726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:20.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 --2- 172.21.15.113:0/4049527885 >> v2:172.21.15.141:3300/0 conn(0x7f7ee815c140 0x7f7ee80726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 >> v1:172.21.15.141:6789/0 conn(0x7f7ee80af780 legacy=0x7f7ee80ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:20.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee80ae590 con 0x7f7ee80622a0 2024-01-15T11:26:20.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f7ee4001670 con 0x7f7ee80622a0 2024-01-15T11:26:20.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 >> v1:172.21.15.113:6789/0 conn(0x7f7ee80622a0 legacy=0x7f7ee80af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:20.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 --2- 172.21.15.113:0/4049527885 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed4003a00 0x7f7ed4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:20.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 --2- 172.21.15.113:0/4049527885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4004850 0x7f7ed4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:20.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 --2- 172.21.15.113:0/4049527885 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7ed4005210 0x7f7ed400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:20.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee80afbf0 con 0x7f7ed4004850 2024-01-15T11:26:20.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed400c0b0 con 0x7f7ed4003a00 2024-01-15T11:26:20.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed400c270 con 0x7f7ed4005210 2024-01-15T11:26:20.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eed489700 1 --2- 172.21.15.113:0/4049527885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4004850 0x7f7ed4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:20.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eedc8a700 1 --2- 172.21.15.113:0/4049527885 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed4003a00 0x7f7ed4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:20.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eee48b700 1 --2- 172.21.15.113:0/4049527885 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7ed4005210 0x7f7ed400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:20.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eed489700 1 -- 172.21.15.113:0/4049527885 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7ed4005210 msgr2=0x7f7ed400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:20.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.737+0000 7f7eed489700 1 --2- 172.21.15.113:0/4049527885 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7ed4005210 0x7f7ed400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eed489700 1 -- 172.21.15.113:0/4049527885 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed4003a00 msgr2=0x7f7ed4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:20.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eed489700 1 --2- 172.21.15.113:0/4049527885 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed4003a00 0x7f7ed4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eed489700 1 -- 172.21.15.113:0/4049527885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed40035b0 con 0x7f7ed4004850 2024-01-15T11:26:20.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eee48b700 1 -- 172.21.15.113:0/4049527885 reap_dead start 2024-01-15T11:26:20.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eed489700 1 --2- 172.21.15.113:0/4049527885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4004850 0x7f7ed4004cd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c210 tx=0x7f7ed4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47628a118f242d61 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:20.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4007c10 con 0x7f7ed4004850 2024-01-15T11:26:20.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee400cdd0 con 0x7f7ed4004850 2024-01-15T11:26:20.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eecc88700 1 -- 172.21.15.113:0/4049527885 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4005ba0 con 0x7f7ed4004850 2024-01-15T11:26:20.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eefeee700 1 -- 172.21.15.113:0/4049527885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4004850 msgr2=0x7f7ed4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:20.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eefeee700 1 --2- 172.21.15.113:0/4049527885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4004850 0x7f7ed4004cd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c210 tx=0x7f7ed4005880 comp rx=0 tx=0).stop 2024-01-15T11:26:20.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eefeee700 1 -- 172.21.15.113:0/4049527885 shutdown_connections 2024-01-15T11:26:20.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eefeee700 1 --2- 172.21.15.113:0/4049527885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4004850 0x7f7ed4004cd0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eefeee700 1 -- 172.21.15.113:0/4049527885 >> 172.21.15.113:0/4049527885 conn(0x7f7ee80b57b0 msgr2=0x7f7ee80b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:20.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eefeee700 1 -- 172.21.15.113:0/4049527885 shutdown_connections 2024-01-15T11:26:20.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.738+0000 7f7eefeee700 1 -- 172.21.15.113:0/4049527885 wait complete. 2024-01-15T11:26:20.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.740+0000 7f7eefeee700 1 Processor -- start 2024-01-15T11:26:20.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.740+0000 7f7eefeee700 1 -- start start 2024-01-15T11:26:20.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.740+0000 7f7eefeee700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed4004850 0x7f7ee81f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:20.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.740+0000 7f7eefeee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ee8154b50 0x7f7ee80cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:20.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.740+0000 7f7eefeee700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7ee815ec40 0x7f7ee80cf490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:20.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.740+0000 7f7eefeee700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee815e820 con 0x7f7ee8154b50 2024-01-15T11:26:20.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.740+0000 7f7eefeee700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee80cb400 con 0x7f7ed4004850 2024-01-15T11:26:20.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.740+0000 7f7eefeee700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee80cb160 con 0x7f7ee815ec40 2024-01-15T11:26:20.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.740+0000 7f7eed489700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ee8154b50 0x7f7ee80cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:20.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.740+0000 7f7eed489700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ee8154b50 0x7f7ee80cac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52174/0 (socket says 172.21.15.113:52174) 2024-01-15T11:26:20.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eed489700 1 -- 172.21.15.113:0/3709011868 learned_addr learned my addr 172.21.15.113:0/3709011868 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:20.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eedc8a700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed4004850 0x7f7ee81f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:20.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eee48b700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7ee815ec40 0x7f7ee80cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:20.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eed489700 1 -- 172.21.15.113:0/3709011868 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7ee815ec40 msgr2=0x7f7ee80cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:20.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eed489700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7ee815ec40 0x7f7ee80cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eed489700 1 -- 172.21.15.113:0/3709011868 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed4004850 msgr2=0x7f7ee81f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:20.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eed489700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed4004850 0x7f7ee81f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eed489700 1 -- 172.21.15.113:0/3709011868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee815de80 con 0x7f7ee8154b50 2024-01-15T11:26:20.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eed489700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ee8154b50 0x7f7ee80cac20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c800 tx=0x7f7ee4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:20.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eda7fc700 1 -- 172.21.15.113:0/3709011868 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4001430 con 0x7f7ee8154b50 2024-01-15T11:26:20.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eda7fc700 1 -- 172.21.15.113:0/3709011868 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee40015d0 con 0x7f7ee8154b50 2024-01-15T11:26:20.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eefeee700 1 -- 172.21.15.113:0/3709011868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee815e160 con 0x7f7ee8154b50 2024-01-15T11:26:20.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eda7fc700 1 -- 172.21.15.113:0/3709011868 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4007310 con 0x7f7ee8154b50 2024-01-15T11:26:20.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.741+0000 7f7eefeee700 1 -- 172.21.15.113:0/3709011868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee815f0c0 con 0x7f7ee8154b50 2024-01-15T11:26:20.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.742+0000 7f7eefeee700 1 -- 172.21.15.113:0/3709011868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee8067f60 con 0x7f7ee8154b50 2024-01-15T11:26:20.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.743+0000 7f7eda7fc700 1 -- 172.21.15.113:0/3709011868 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7ee4007470 con 0x7f7ee8154b50 2024-01-15T11:26:20.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.743+0000 7f7eda7fc700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7ecc06b350 0x7f7ecc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:20.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.743+0000 7f7eda7fc700 1 -- 172.21.15.113:0/3709011868 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ee408b080 con 0x7f7ee8154b50 2024-01-15T11:26:20.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.743+0000 7f7eedc8a700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7ecc06b350 0x7f7ecc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:20.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.743+0000 7f7eedc8a700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7ecc06b350 0x7f7ecc06d810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009ce0 tx=0x7f7edc009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:20.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.747+0000 7f7eda7fc700 1 -- 172.21.15.113:0/3709011868 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ee4056020 con 0x7f7ee8154b50 2024-01-15T11:26:20.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.965+0000 7f7eefeee700 1 -- 172.21.15.113:0/3709011868 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ee80adc80 con 0x7f7ecc06b350 2024-01-15T11:26:20.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.967+0000 7f7eda7fc700 1 -- 172.21.15.113:0/3709011868 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7edc019070 con 0x7f7ecc06b350 2024-01-15T11:26:20.966 INFO:teuthology.orchestra.run.smithi113.stdout:[ 2024-01-15T11:26:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: "name": "cephfs" 2024-01-15T11:26:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:20.966 INFO:teuthology.orchestra.run.smithi113.stdout:] 2024-01-15T11:26:20.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.969+0000 7f7eefeee700 1 -- 172.21.15.113:0/3709011868 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7ecc06b350 msgr2=0x7f7ecc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:20.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.969+0000 7f7eefeee700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7ecc06b350 0x7f7ecc06d810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009ce0 tx=0x7f7edc009210 comp rx=0 tx=0).stop 2024-01-15T11:26:20.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.969+0000 7f7eefeee700 1 -- 172.21.15.113:0/3709011868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ee8154b50 msgr2=0x7f7ee80cac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:20.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.969+0000 7f7eefeee700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ee8154b50 0x7f7ee80cac20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c800 tx=0x7f7ee4001a00 comp rx=0 tx=0).stop 2024-01-15T11:26:20.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.969+0000 7f7eefeee700 1 -- 172.21.15.113:0/3709011868 shutdown_connections 2024-01-15T11:26:20.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.969+0000 7f7eefeee700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f7ecc06b350 0x7f7ecc06d810 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.969+0000 7f7eefeee700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f7ee815ec40 0x7f7ee80cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.969+0000 7f7eefeee700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7ed4004850 0x7f7ee81f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.969+0000 7f7eefeee700 1 --2- 172.21.15.113:0/3709011868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ee8154b50 0x7f7ee80cac20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:20.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.969+0000 7f7eefeee700 1 -- 172.21.15.113:0/3709011868 >> 172.21.15.113:0/3709011868 conn(0x7f7ee80b57b0 msgr2=0x7f7ee815b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:20.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.969+0000 7f7eefeee700 1 -- 172.21.15.113:0/3709011868 shutdown_connections 2024-01-15T11:26:20.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:20.969+0000 7f7eefeee700 1 -- 172.21.15.113:0/3709011868 wait complete. 2024-01-15T11:26:20.980 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T11:26:21.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.105+0000 7f649907d700 1 Processor -- start 2024-01-15T11:26:21.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.105+0000 7f649907d700 1 -- start start 2024-01-15T11:26:21.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.106+0000 7f649907d700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f649415f8d0 0x7f64940b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.106+0000 7f649907d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f64940d03a0 0x7f64940ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.106+0000 7f649907d700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64940b3860 con 0x7f64940cffc0 2024-01-15T11:26:21.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.106+0000 7f649907d700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64940b2520 con 0x7f64940d03a0 2024-01-15T11:26:21.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.106+0000 7f649907d700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64940b2800 con 0x7f649415f8d0 2024-01-15T11:26:21.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.106+0000 7f649359e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f64940d03a0 0x7f64940ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.106+0000 7f649359e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f64940d03a0 0x7f64940ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52178/0 (socket says 172.21.15.113:52178) 2024-01-15T11:26:21.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.106+0000 7f649359e700 1 -- 172.21.15.113:0/987001413 learned_addr learned my addr 172.21.15.113:0/987001413 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:21.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649359e700 1 -- 172.21.15.113:0/987001413 >> v2:172.21.15.141:3300/0 conn(0x7f649415f8d0 msgr2=0x7f64940b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:21.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.106+0000 7f649259c700 1 --2- 172.21.15.113:0/987001413 >> v2:172.21.15.141:3300/0 conn(0x7f649415f8d0 0x7f64940b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649359e700 1 --2- 172.21.15.113:0/987001413 >> v2:172.21.15.141:3300/0 conn(0x7f649415f8d0 0x7f64940b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649359e700 1 -- 172.21.15.113:0/987001413 >> v1:172.21.15.141:6789/0 conn(0x7f64940cffc0 legacy=0x7f649415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:21.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649359e700 1 -- 172.21.15.113:0/987001413 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64940b2ae0 con 0x7f64940d03a0 2024-01-15T11:26:21.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649359e700 1 --2- 172.21.15.113:0/987001413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64940d03a0 0x7f64940ccd10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6494072330 tx=0x7f648c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c41a1ee20d6ed9d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:21.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f6491d9b700 1 -- 172.21.15.113:0/987001413 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c00e070 con 0x7f64940d03a0 2024-01-15T11:26:21.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f6491d9b700 1 -- 172.21.15.113:0/987001413 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648c003a20 con 0x7f64940d03a0 2024-01-15T11:26:21.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f6491d9b700 1 -- 172.21.15.113:0/987001413 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c003dd0 con 0x7f64940d03a0 2024-01-15T11:26:21.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649907d700 1 -- 172.21.15.113:0/987001413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64940d03a0 msgr2=0x7f64940ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649907d700 1 --2- 172.21.15.113:0/987001413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64940d03a0 0x7f64940ccd10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6494072330 tx=0x7f648c00b330 comp rx=0 tx=0).stop 2024-01-15T11:26:21.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649907d700 1 -- 172.21.15.113:0/987001413 shutdown_connections 2024-01-15T11:26:21.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649907d700 1 --2- 172.21.15.113:0/987001413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64940d03a0 0x7f64940ccd10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649907d700 1 --2- 172.21.15.113:0/987001413 >> v2:172.21.15.141:3300/0 conn(0x7f649415f8d0 0x7f64940b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649907d700 1 -- 172.21.15.113:0/987001413 >> 172.21.15.113:0/987001413 conn(0x7f64940d1e50 msgr2=0x7f64940d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:21.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649907d700 1 -- 172.21.15.113:0/987001413 shutdown_connections 2024-01-15T11:26:21.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.107+0000 7f649907d700 1 -- 172.21.15.113:0/987001413 wait complete. 2024-01-15T11:26:21.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.109+0000 7f649907d700 1 Processor -- start 2024-01-15T11:26:21.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.109+0000 7f649907d700 1 -- start start 2024-01-15T11:26:21.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649907d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64940cffc0 0x7f64941e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649907d700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f64940d03a0 0x7f64941e4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649907d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f649415f8d0 0x7f649415b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649907d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64941e7a70 con 0x7f649415f8d0 2024-01-15T11:26:21.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649907d700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64941e7790 con 0x7f64940cffc0 2024-01-15T11:26:21.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649907d700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64940b6550 con 0x7f64940d03a0 2024-01-15T11:26:21.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649359e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f649415f8d0 0x7f649415b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649359e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f649415f8d0 0x7f649415b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52186/0 (socket says 172.21.15.113:52186) 2024-01-15T11:26:21.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649359e700 1 -- 172.21.15.113:0/3027956446 learned_addr learned my addr 172.21.15.113:0/3027956446 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:21.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649259c700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f64940d03a0 0x7f64941e4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f6492d9d700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64940cffc0 0x7f64941e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649359e700 1 -- 172.21.15.113:0/3027956446 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f64940d03a0 msgr2=0x7f64941e4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649359e700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f64940d03a0 0x7f64941e4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649359e700 1 -- 172.21.15.113:0/3027956446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64940cffc0 msgr2=0x7f64941e6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649359e700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64940cffc0 0x7f64941e6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649359e700 1 -- 172.21.15.113:0/3027956446 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c009b30 con 0x7f649415f8d0 2024-01-15T11:26:21.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649259c700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f64940d03a0 0x7f64941e4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:21.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649359e700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f649415f8d0 0x7f649415b410 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f648c015040 tx=0x7f648c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:21.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f647b7fe700 1 -- 172.21.15.113:0/3027956446 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c004440 con 0x7f649415f8d0 2024-01-15T11:26:21.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f647b7fe700 1 -- 172.21.15.113:0/3027956446 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648c0045a0 con 0x7f649415f8d0 2024-01-15T11:26:21.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f649907d700 1 -- 172.21.15.113:0/3027956446 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64940b6830 con 0x7f649415f8d0 2024-01-15T11:26:21.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.110+0000 7f647b7fe700 1 -- 172.21.15.113:0/3027956446 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c004950 con 0x7f649415f8d0 2024-01-15T11:26:21.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.111+0000 7f649907d700 1 -- 172.21.15.113:0/3027956446 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64941e7480 con 0x7f649415f8d0 2024-01-15T11:26:21.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.111+0000 7f649907d700 1 -- 172.21.15.113:0/3027956446 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64940b63b0 con 0x7f649415f8d0 2024-01-15T11:26:21.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.112+0000 7f647b7fe700 1 -- 172.21.15.113:0/3027956446 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f648c02e430 con 0x7f649415f8d0 2024-01-15T11:26:21.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.113+0000 7f647b7fe700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f648006b350 0x7f648006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.113+0000 7f647b7fe700 1 -- 172.21.15.113:0/3027956446 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f648c08b080 con 0x7f649415f8d0 2024-01-15T11:26:21.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.113+0000 7f6492d9d700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f648006b350 0x7f648006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.113+0000 7f6492d9d700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f648006b350 0x7f648006d810 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6488007fd0 tx=0x7f6488007ec0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:21.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.117+0000 7f647b7fe700 1 -- 172.21.15.113:0/3027956446 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f648c059af0 con 0x7f649415f8d0 2024-01-15T11:26:21.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.307+0000 7f649907d700 1 -- 172.21.15.113:0/3027956446 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f64940b3860 con 0x7f649415f8d0 2024-01-15T11:26:21.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.307+0000 7f647b7fe700 1 -- 172.21.15.113:0/3027956446 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f64940b3860 con 0x7f649415f8d0 2024-01-15T11:26:21.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.309+0000 7f649907d700 1 -- 172.21.15.113:0/3027956446 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f648006b350 msgr2=0x7f648006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.309+0000 7f649907d700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f648006b350 0x7f648006d810 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6488007fd0 tx=0x7f6488007ec0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.309+0000 7f649907d700 1 -- 172.21.15.113:0/3027956446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f649415f8d0 msgr2=0x7f649415b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.309+0000 7f649907d700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f649415f8d0 0x7f649415b410 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f648c015040 tx=0x7f648c004ad0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.309+0000 7f649907d700 1 -- 172.21.15.113:0/3027956446 shutdown_connections 2024-01-15T11:26:21.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.309+0000 7f649907d700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f648006b350 0x7f648006d810 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.309+0000 7f649907d700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f649415f8d0 0x7f649415b410 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.309+0000 7f649907d700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64940cffc0 0x7f64941e6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.309+0000 7f649907d700 1 --2- 172.21.15.113:0/3027956446 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f64940d03a0 0x7f64941e4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.309+0000 7f649907d700 1 -- 172.21.15.113:0/3027956446 >> 172.21.15.113:0/3027956446 conn(0x7f64940d1e50 msgr2=0x7f64940d2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:21.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.309+0000 7f649907d700 1 -- 172.21.15.113:0/3027956446 shutdown_connections 2024-01-15T11:26:21.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.309+0000 7f649907d700 1 -- 172.21.15.113:0/3027956446 wait complete. 2024-01-15T11:26:21.321 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T11:26:21.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.446+0000 7fa4c6fd3700 1 Processor -- start 2024-01-15T11:26:21.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.446+0000 7fa4c6fd3700 1 -- start start 2024-01-15T11:26:21.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4c6fd3700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa4c01622b0 0x7fa4c006c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4c6fd3700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4c0162820 0x7fa4c006caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4c6fd3700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c006bed0 con 0x7fa4c015df20 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4c6fd3700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c006cfe0 con 0x7fa4c01622b0 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4c6fd3700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c006bba0 con 0x7fa4c0162820 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4bffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa4c01622b0 0x7fa4c006c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4bffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa4c01622b0 0x7fa4c006c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52194/0 (socket says 172.21.15.113:52194) 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4bffff700 1 -- 172.21.15.113:0/1478854834 learned_addr learned my addr 172.21.15.113:0/1478854834 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4c5570700 1 --2- 172.21.15.113:0/1478854834 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4c0162820 0x7fa4c006caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4bffff700 1 -- 172.21.15.113:0/1478854834 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4c0162820 msgr2=0x7fa4c006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4bffff700 1 --2- 172.21.15.113:0/1478854834 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4c0162820 0x7fa4c006caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4bffff700 1 -- 172.21.15.113:0/1478854834 >> v1:172.21.15.141:6789/0 conn(0x7fa4c015df20 legacy=0x7fa4c0161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4bffff700 1 -- 172.21.15.113:0/1478854834 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c0070060 con 0x7fa4c01622b0 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.447+0000 7fa4bffff700 1 --2- 172.21.15.113:0/1478854834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c01622b0 0x7fa4c006c360 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0098f0 tx=0x7fa4ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3909d136868085 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.448+0000 7fa4bf7fe700 1 -- 172.21.15.113:0/1478854834 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac00b8a0 con 0x7fa4c01622b0 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.448+0000 7fa4bf7fe700 1 -- 172.21.15.113:0/1478854834 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac00ba00 con 0x7fa4c01622b0 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.448+0000 7fa4bf7fe700 1 -- 172.21.15.113:0/1478854834 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac00bd90 con 0x7fa4c01622b0 2024-01-15T11:26:21.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.448+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/1478854834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c01622b0 msgr2=0x7fa4c006c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.448+0000 7fa4c6fd3700 1 --2- 172.21.15.113:0/1478854834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c01622b0 0x7fa4c006c360 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0098f0 tx=0x7fa4ac009c00 comp rx=0 tx=0).stop 2024-01-15T11:26:21.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.448+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/1478854834 shutdown_connections 2024-01-15T11:26:21.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.448+0000 7fa4c6fd3700 1 --2- 172.21.15.113:0/1478854834 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4c0162820 0x7fa4c006caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.448+0000 7fa4c6fd3700 1 --2- 172.21.15.113:0/1478854834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c01622b0 0x7fa4c006c360 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.448+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/1478854834 >> 172.21.15.113:0/1478854834 conn(0x7fa4c006dc80 msgr2=0x7fa4c009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:21.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.448+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/1478854834 shutdown_connections 2024-01-15T11:26:21.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.448+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/1478854834 wait complete. 2024-01-15T11:26:21.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.450+0000 7fa4c6fd3700 1 Processor -- start 2024-01-15T11:26:21.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.450+0000 7fa4c6fd3700 1 -- start start 2024-01-15T11:26:21.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4c6fd3700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4c015df20 0x7fa4c014cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4c6fd3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c01622b0 0x7fa4c00b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4c6fd3700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4c0162820 0x7fa4c00b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4c6fd3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c01f7480 con 0x7fa4c01622b0 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4c6fd3700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00cbe30 con 0x7fa4c015df20 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4c6fd3700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00c9210 con 0x7fa4c0162820 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4bffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c01622b0 0x7fa4c00b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4bffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c01622b0 0x7fa4c00b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52200/0 (socket says 172.21.15.113:52200) 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4bffff700 1 -- 172.21.15.113:0/123343758 learned_addr learned my addr 172.21.15.113:0/123343758 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4c5570700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4c0162820 0x7fa4c00b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4bffff700 1 -- 172.21.15.113:0/123343758 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4c0162820 msgr2=0x7fa4c00b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4bffff700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4c0162820 0x7fa4c00b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4bffff700 1 -- 172.21.15.113:0/123343758 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4c015df20 msgr2=0x7fa4c014cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4c4d6f700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4c015df20 0x7fa4c014cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4bffff700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4c015df20 0x7fa4c014cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4bffff700 1 -- 172.21.15.113:0/123343758 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac009550 con 0x7fa4c01622b0 2024-01-15T11:26:21.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4c5570700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4c0162820 0x7fa4c00b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:21.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4c4d6f700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4c015df20 0x7fa4c014cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:21.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.451+0000 7fa4bffff700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c01622b0 0x7fa4c00b2700 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009c40 tx=0x7fa4ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:21.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.452+0000 7fa4bd7fa700 1 -- 172.21.15.113:0/123343758 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac0036a0 con 0x7fa4c01622b0 2024-01-15T11:26:21.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.452+0000 7fa4bd7fa700 1 -- 172.21.15.113:0/123343758 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac01e070 con 0x7fa4c01622b0 2024-01-15T11:26:21.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.452+0000 7fa4bd7fa700 1 -- 172.21.15.113:0/123343758 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac003a60 con 0x7fa4c01622b0 2024-01-15T11:26:21.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.452+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/123343758 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c014d1c0 con 0x7fa4c01622b0 2024-01-15T11:26:21.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.452+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/123343758 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c00cb070 con 0x7fa4c01622b0 2024-01-15T11:26:21.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.453+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/123343758 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c0067f60 con 0x7fa4c01622b0 2024-01-15T11:26:21.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.453+0000 7fa4bd7fa700 1 -- 172.21.15.113:0/123343758 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa4ac003bc0 con 0x7fa4c01622b0 2024-01-15T11:26:21.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.453+0000 7fa4bd7fa700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa4a406b350 0x7fa4a406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.453+0000 7fa4bd7fa700 1 -- 172.21.15.113:0/123343758 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4ac020070 con 0x7fa4c01622b0 2024-01-15T11:26:21.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.453+0000 7fa4c4d6f700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa4a406b350 0x7fa4a406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.454+0000 7fa4c4d6f700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa4a406b350 0x7fa4a406d810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa4c009ff50 tx=0x7fa4b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:21.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.458+0000 7fa4bd7fa700 1 -- 172.21.15.113:0/123343758 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4ac055ec0 con 0x7fa4c01622b0 2024-01-15T11:26:21.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.678+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/123343758 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4c00d2a60 con 0x7fa4a406b350 2024-01-15T11:26:21.681 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:21.682+0000 7fa4c94a7700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:21.681 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:21.682+0000 7fa4c94a7700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:21.681 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:21.682+0000 7fa4c94a7700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:21.681 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:21.682+0000 7fa4c94a7700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:21.681 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:21.682+0000 7fa4c94a7700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:21.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.720+0000 7fa4bd7fa700 1 -- 172.21.15.113:0/123343758 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa4c00d2a60 con 0x7fa4a406b350 2024-01-15T11:26:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.722+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/123343758 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa4a406b350 msgr2=0x7fa4a406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.722+0000 7fa4c6fd3700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa4a406b350 0x7fa4a406d810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa4c009ff50 tx=0x7fa4b4009040 comp rx=0 tx=0).stop 2024-01-15T11:26:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.722+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/123343758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c01622b0 msgr2=0x7fa4c00b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.722+0000 7fa4c6fd3700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c01622b0 0x7fa4c00b2700 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009c40 tx=0x7fa4ac004750 comp rx=0 tx=0).stop 2024-01-15T11:26:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.722+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/123343758 shutdown_connections 2024-01-15T11:26:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.722+0000 7fa4c6fd3700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa4a406b350 0x7fa4a406d810 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.722+0000 7fa4c6fd3700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa4c0162820 0x7fa4c00b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.722+0000 7fa4c6fd3700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa4c015df20 0x7fa4c014cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.722+0000 7fa4c6fd3700 1 --2- 172.21.15.113:0/123343758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c01622b0 0x7fa4c00b2700 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.722+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/123343758 >> 172.21.15.113:0/123343758 conn(0x7fa4c006dc80 msgr2=0x7fa4c00b0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.722+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/123343758 shutdown_connections 2024-01-15T11:26:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.722+0000 7fa4c6fd3700 1 -- 172.21.15.113:0/123343758 wait complete. 2024-01-15T11:26:21.733 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T11:26:21.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.857+0000 7fb40c04d700 1 Processor -- start 2024-01-15T11:26:21.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.858+0000 7fb40c04d700 1 -- start start 2024-01-15T11:26:21.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.858+0000 7fb40c04d700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fb4040cfb10 0x7fb4040d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.858+0000 7fb40c04d700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb4040b4550 0x7fb40415e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.858+0000 7fb40c04d700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4040b3430 con 0x7fb40415a3c0 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.858+0000 7fb40c04d700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb40415ef00 con 0x7fb4040cfb10 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.858+0000 7fb40c04d700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4040cff40 con 0x7fb4040b4550 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb409de9700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb40415a3c0 0x7fb4040b3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56516/0 (socket says 172.21.15.113:56516) 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb409de9700 1 -- 172.21.15.113:0/2553251861 learned_addr learned my addr 172.21.15.113:0/2553251861 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb4095e8700 1 --2- 172.21.15.113:0/2553251861 >> v2:172.21.15.141:3300/0 conn(0x7fb4040cfb10 0x7fb4040d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb40a5ea700 1 --2- 172.21.15.113:0/2553251861 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb4040b4550 0x7fb40415e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb408de7700 1 -- 172.21.15.113:0/2553251861 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646099424 0 0) 0x7fb4040b3430 con 0x7fb40415a3c0 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb408de7700 1 -- 172.21.15.113:0/2553251861 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3f00035b0 con 0x7fb40415a3c0 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb4095e8700 1 -- 172.21.15.113:0/2553251861 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb4040b4550 msgr2=0x7fb40415e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb4095e8700 1 --2- 172.21.15.113:0/2553251861 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb4040b4550 0x7fb40415e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb4095e8700 1 -- 172.21.15.113:0/2553251861 >> v1:172.21.15.113:6789/0 conn(0x7fb40415a3c0 legacy=0x7fb4040b3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb4095e8700 1 -- 172.21.15.113:0/2553251861 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4040b36f0 con 0x7fb4040cfb10 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb4095e8700 1 --2- 172.21.15.113:0/2553251861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4040cfb10 0x7fb4040d0a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb400009840 tx=0x7fb400009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15eb5acbc57efb1b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:21.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb408de7700 1 -- 172.21.15.113:0/2553251861 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400011830 con 0x7fb4040cfb10 2024-01-15T11:26:21.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb408de7700 1 -- 172.21.15.113:0/2553251861 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb400011990 con 0x7fb4040cfb10 2024-01-15T11:26:21.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.859+0000 7fb408de7700 1 -- 172.21.15.113:0/2553251861 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400011d00 con 0x7fb4040cfb10 2024-01-15T11:26:21.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.860+0000 7fb40c04d700 1 -- 172.21.15.113:0/2553251861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4040cfb10 msgr2=0x7fb4040d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.860+0000 7fb40c04d700 1 --2- 172.21.15.113:0/2553251861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4040cfb10 0x7fb4040d0a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb400009840 tx=0x7fb400009b50 comp rx=0 tx=0).stop 2024-01-15T11:26:21.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.860+0000 7fb40c04d700 1 -- 172.21.15.113:0/2553251861 shutdown_connections 2024-01-15T11:26:21.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.860+0000 7fb40c04d700 1 --2- 172.21.15.113:0/2553251861 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb4040b4550 0x7fb40415e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.860+0000 7fb40c04d700 1 --2- 172.21.15.113:0/2553251861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4040cfb10 0x7fb4040d0a00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.860+0000 7fb40c04d700 1 -- 172.21.15.113:0/2553251861 >> 172.21.15.113:0/2553251861 conn(0x7fb4040743b0 msgr2=0x7fb40406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:21.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.860+0000 7fb40c04d700 1 -- 172.21.15.113:0/2553251861 shutdown_connections 2024-01-15T11:26:21.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.860+0000 7fb40c04d700 1 -- 172.21.15.113:0/2553251861 wait complete. 2024-01-15T11:26:21.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.862+0000 7fb40c04d700 1 Processor -- start 2024-01-15T11:26:21.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.862+0000 7fb40c04d700 1 -- start start 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.862+0000 7fb40c04d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4040b4550 0x7fb40415ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.862+0000 7fb40c04d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb40415a3c0 0x7fb404152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.862+0000 7fb40c04d700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb40414d860 0x7fb4040cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.862+0000 7fb40c04d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb404153a30 con 0x7fb4040b4550 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.862+0000 7fb40c04d700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb404153470 con 0x7fb40415a3c0 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.862+0000 7fb40c04d700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb404153750 con 0x7fb40414d860 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb409de9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4040b4550 0x7fb40415ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb409de9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4040b4550 0x7fb40415ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52232/0 (socket says 172.21.15.113:52232) 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb409de9700 1 -- 172.21.15.113:0/1593203859 learned_addr learned my addr 172.21.15.113:0/1593203859 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb40a5ea700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb40414d860 0x7fb4040cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb4095e8700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb40415a3c0 0x7fb404152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb409de9700 1 -- 172.21.15.113:0/1593203859 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb40414d860 msgr2=0x7fb4040cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb409de9700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb40414d860 0x7fb4040cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb409de9700 1 -- 172.21.15.113:0/1593203859 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb40415a3c0 msgr2=0x7fb404152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:21.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb409de9700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb40415a3c0 0x7fb404152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:21.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb409de9700 1 -- 172.21.15.113:0/1593203859 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4000094a0 con 0x7fb4040b4550 2024-01-15T11:26:21.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb409de9700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4040b4550 0x7fb40415ac30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400c420 tx=0x7fb3f400c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:21.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb3fa7fc700 1 -- 172.21.15.113:0/1593203859 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4005d70 con 0x7fb4040b4550 2024-01-15T11:26:21.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb3fa7fc700 1 -- 172.21.15.113:0/1593203859 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f400ce30 con 0x7fb4040b4550 2024-01-15T11:26:21.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb40c04d700 1 -- 172.21.15.113:0/1593203859 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4040b0fc0 con 0x7fb4040b4550 2024-01-15T11:26:21.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb3fa7fc700 1 -- 172.21.15.113:0/1593203859 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40072d0 con 0x7fb4040b4550 2024-01-15T11:26:21.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.863+0000 7fb40c04d700 1 -- 172.21.15.113:0/1593203859 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40414dd10 con 0x7fb4040b4550 2024-01-15T11:26:21.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.864+0000 7fb40c04d700 1 -- 172.21.15.113:0/1593203859 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4040b3cb0 con 0x7fb4040b4550 2024-01-15T11:26:21.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.865+0000 7fb3fa7fc700 1 -- 172.21.15.113:0/1593203859 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb3f4007430 con 0x7fb4040b4550 2024-01-15T11:26:21.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.865+0000 7fb3fa7fc700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb3e806b380 0x7fb3e806d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:21.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.865+0000 7fb3fa7fc700 1 -- 172.21.15.113:0/1593203859 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3f4088e90 con 0x7fb4040b4550 2024-01-15T11:26:21.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.865+0000 7fb4095e8700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb3e806b380 0x7fb3e806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:21.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.866+0000 7fb4095e8700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb3e806b380 0x7fb3e806d840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb400000c00 tx=0x7fb400008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:21.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:21.870+0000 7fb3fa7fc700 1 -- 172.21.15.113:0/1593203859 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3f4053ca0 con 0x7fb4040b4550 2024-01-15T11:26:22.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.093+0000 7fb40c04d700 1 -- 172.21.15.113:0/1593203859 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4040d04e0 con 0x7fb3e806b380 2024-01-15T11:26:22.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.098+0000 7fb3fa7fc700 1 -- 172.21.15.113:0/1593203859 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fb400011700 con 0x7fb3e806b380 2024-01-15T11:26:22.097 INFO:teuthology.orchestra.run.smithi113.stdout:/volumes/_nogroup/sv1/495e2b63-0db2-4f7a-b1b1-d2919dd37cc5 2024-01-15T11:26:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.100+0000 7fb40c04d700 1 -- 172.21.15.113:0/1593203859 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb3e806b380 msgr2=0x7fb3e806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.100+0000 7fb40c04d700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb3e806b380 0x7fb3e806d840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb400000c00 tx=0x7fb400008030 comp rx=0 tx=0).stop 2024-01-15T11:26:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.100+0000 7fb40c04d700 1 -- 172.21.15.113:0/1593203859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4040b4550 msgr2=0x7fb40415ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.100+0000 7fb40c04d700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4040b4550 0x7fb40415ac30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400c420 tx=0x7fb3f400c730 comp rx=0 tx=0).stop 2024-01-15T11:26:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.100+0000 7fb40c04d700 1 -- 172.21.15.113:0/1593203859 shutdown_connections 2024-01-15T11:26:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.100+0000 7fb40c04d700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb3e806b380 0x7fb3e806d840 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.100+0000 7fb40c04d700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb40414d860 0x7fb4040cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.100+0000 7fb40c04d700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4040b4550 0x7fb40415ac30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.100+0000 7fb40c04d700 1 --2- 172.21.15.113:0/1593203859 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb40415a3c0 0x7fb404152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.100+0000 7fb40c04d700 1 -- 172.21.15.113:0/1593203859 >> 172.21.15.113:0/1593203859 conn(0x7fb4040743b0 msgr2=0x7fb40406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.100+0000 7fb40c04d700 1 -- 172.21.15.113:0/1593203859 shutdown_connections 2024-01-15T11:26:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:22.100+0000 7fb40c04d700 1 -- 172.21.15.113:0/1593203859 wait complete. 2024-01-15T11:26:22.111 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/495e2b63-0db2-4f7a-b1b1-d2919dd37cc5 2024-01-15T11:26:22.111 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:26:22.112 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:26:22.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:22.129 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/495e2b63-0db2-4f7a-b1b1-d2919dd37cc5/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T11:26:27.692 INFO:teuthology.orchestra.run.smithi113.stderr:1024+0 records in 2024-01-15T11:26:27.692 INFO:teuthology.orchestra.run.smithi113.stderr:1024+0 records out 2024-01-15T11:26:27.693 INFO:teuthology.orchestra.run.smithi113.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.49769 s, 195 MB/s 2024-01-15T11:26:27.693 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:26:27.693 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:26:27.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:27.712 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/495e2b63-0db2-4f7a-b1b1-d2919dd37cc5/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T11:26:33.247 INFO:teuthology.orchestra.run.smithi113.stderr:1024+0 records in 2024-01-15T11:26:33.247 INFO:teuthology.orchestra.run.smithi113.stderr:1024+0 records out 2024-01-15T11:26:33.247 INFO:teuthology.orchestra.run.smithi113.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.47608 s, 196 MB/s 2024-01-15T11:26:33.247 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:26:33.247 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:26:33.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:33.265 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/495e2b63-0db2-4f7a-b1b1-d2919dd37cc5/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T11:26:38.731 INFO:teuthology.orchestra.run.smithi113.stderr:1024+0 records in 2024-01-15T11:26:38.732 INFO:teuthology.orchestra.run.smithi113.stderr:1024+0 records out 2024-01-15T11:26:38.732 INFO:teuthology.orchestra.run.smithi113.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.40644 s, 199 MB/s 2024-01-15T11:26:38.732 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-15T11:26:38.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.859+0000 7f2960564700 1 Processor -- start 2024-01-15T11:26:38.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.859+0000 7f2960564700 1 -- start start 2024-01-15T11:26:38.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.860+0000 7f2960564700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2958151620 0x7f29580cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:38.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.860+0000 7f2960564700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29580b3f60 con 0x7f29580b3470 2024-01-15T11:26:38.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.860+0000 7f2960564700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295806d190 con 0x7f295815df60 2024-01-15T11:26:38.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.860+0000 7f2960564700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f295806d750 con 0x7f2958151620 2024-01-15T11:26:38.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.860+0000 7f295daff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f29580b3470 0x7f2958070b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:33722/0 (socket says 172.21.15.113:33722) 2024-01-15T11:26:38.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.860+0000 7f295daff700 1 -- 172.21.15.113:0/2416328315 learned_addr learned my addr 172.21.15.113:0/2416328315 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:38.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.860+0000 7f295e300700 1 --2- 172.21.15.113:0/2416328315 >> v2:172.21.15.141:3300/0 conn(0x7f2958151620 0x7f29580cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:38.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.861+0000 7f295d2fe700 1 -- 172.21.15.113:0/2416328315 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110315123 0 0) 0x7f29580b3f60 con 0x7f29580b3470 2024-01-15T11:26:38.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.861+0000 7f295d2fe700 1 -- 172.21.15.113:0/2416328315 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29440035b0 con 0x7f29580b3470 2024-01-15T11:26:38.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.861+0000 7f295d2fe700 1 -- 172.21.15.113:0/2416328315 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 965540555 0 0) 0x7f29440035b0 con 0x7f29580b3470 2024-01-15T11:26:38.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.861+0000 7f295e300700 1 -- 172.21.15.113:0/2416328315 >> v1:172.21.15.141:6789/0 conn(0x7f295815df60 legacy=0x7f295806dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:38.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.861+0000 7f295e300700 1 -- 172.21.15.113:0/2416328315 >> v1:172.21.15.113:6789/0 conn(0x7f29580b3470 legacy=0x7f2958070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:38.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.861+0000 7f295e300700 1 -- 172.21.15.113:0/2416328315 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2958071230 con 0x7f2958151620 2024-01-15T11:26:38.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.861+0000 7f295e300700 1 --2- 172.21.15.113:0/2416328315 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2958151620 0x7f29580cf230 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2948009840 tx=0x7f2948009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97c1f1781ca29be5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:38.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.861+0000 7f295d2fe700 1 -- 172.21.15.113:0/2416328315 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2954001670 con 0x7f2958151620 2024-01-15T11:26:38.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.861+0000 7f295d2fe700 1 -- 172.21.15.113:0/2416328315 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2948004030 con 0x7f2958151620 2024-01-15T11:26:38.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.861+0000 7f295d2fe700 1 -- 172.21.15.113:0/2416328315 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29480043a0 con 0x7f2958151620 2024-01-15T11:26:38.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.862+0000 7f2960564700 1 -- 172.21.15.113:0/2416328315 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2958151620 msgr2=0x7f29580cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:38.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.862+0000 7f2960564700 1 --2- 172.21.15.113:0/2416328315 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2958151620 0x7f29580cf230 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2948009840 tx=0x7f2948009b50 comp rx=0 tx=0).stop 2024-01-15T11:26:38.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.862+0000 7f2960564700 1 -- 172.21.15.113:0/2416328315 shutdown_connections 2024-01-15T11:26:38.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.862+0000 7f2960564700 1 --2- 172.21.15.113:0/2416328315 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2958151620 0x7f29580cf230 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:38.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.862+0000 7f2960564700 1 -- 172.21.15.113:0/2416328315 >> 172.21.15.113:0/2416328315 conn(0x7f295806b410 msgr2=0x7f295806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:38.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.862+0000 7f2960564700 1 -- 172.21.15.113:0/2416328315 shutdown_connections 2024-01-15T11:26:38.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.862+0000 7f2960564700 1 -- 172.21.15.113:0/2416328315 wait complete. 2024-01-15T11:26:38.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.863+0000 7f2960564700 1 Processor -- start 2024-01-15T11:26:38.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.863+0000 7f2960564700 1 -- start start 2024-01-15T11:26:38.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f2960564700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29580b3470 0x7f29581e5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:38.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f2960564700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2958151620 0x7f2958146300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:38.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f2960564700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f295815df60 0x7f2958146840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:38.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f2960564700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29581e4dd0 con 0x7f29580b3470 2024-01-15T11:26:38.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f2960564700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29580cd3a0 con 0x7f295815df60 2024-01-15T11:26:38.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f2960564700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29581e4fe0 con 0x7f2958151620 2024-01-15T11:26:38.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f295e300700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29580b3470 0x7f29581e5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:38.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f295e300700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29580b3470 0x7f29581e5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54028/0 (socket says 172.21.15.113:54028) 2024-01-15T11:26:38.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f295e300700 1 -- 172.21.15.113:0/3445874801 learned_addr learned my addr 172.21.15.113:0/3445874801 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:38.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f295daff700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2958151620 0x7f2958146300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:38.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f295eb01700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f295815df60 0x7f2958146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:38.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f295e300700 1 -- 172.21.15.113:0/3445874801 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2958151620 msgr2=0x7f2958146300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:38.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f295e300700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2958151620 0x7f2958146300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:38.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f295e300700 1 -- 172.21.15.113:0/3445874801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f295815df60 msgr2=0x7f2958146840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:38.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f295e300700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f295815df60 0x7f2958146840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:38.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f295e300700 1 -- 172.21.15.113:0/3445874801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29480094a0 con 0x7f29580b3470 2024-01-15T11:26:38.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.864+0000 7f295eb01700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f295815df60 0x7f2958146840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:38.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.865+0000 7f295e300700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29580b3470 0x7f29581e5ac0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f29580d0a20 tx=0x7f2948004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:38.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.865+0000 7f294effd700 1 -- 172.21.15.113:0/3445874801 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2948010790 con 0x7f29580b3470 2024-01-15T11:26:38.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.865+0000 7f2960564700 1 -- 172.21.15.113:0/3445874801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29581e52c0 con 0x7f29580b3470 2024-01-15T11:26:38.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.865+0000 7f294effd700 1 -- 172.21.15.113:0/3445874801 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2948010910 con 0x7f29580b3470 2024-01-15T11:26:38.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.865+0000 7f294effd700 1 -- 172.21.15.113:0/3445874801 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2948010cc0 con 0x7f29580b3470 2024-01-15T11:26:38.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.865+0000 7f2960564700 1 -- 172.21.15.113:0/3445874801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29581458f0 con 0x7f29580b3470 2024-01-15T11:26:38.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.866+0000 7f2960564700 1 -- 172.21.15.113:0/3445874801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29581e57e0 con 0x7f29580b3470 2024-01-15T11:26:38.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.866+0000 7f294effd700 1 -- 172.21.15.113:0/3445874801 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f294801c420 con 0x7f29580b3470 2024-01-15T11:26:38.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.866+0000 7f294effd700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f293c06b3e0 0x7f293c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:38.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.867+0000 7f294effd700 1 -- 172.21.15.113:0/3445874801 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f294808b510 con 0x7f29580b3470 2024-01-15T11:26:38.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.867+0000 7f295daff700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f293c06b3e0 0x7f293c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:38.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.867+0000 7f295daff700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f293c06b3e0 0x7f293c06d8a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2954005a70 tx=0x7f2954001c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:38.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:38.871+0000 7f294effd700 1 -- 172.21.15.113:0/3445874801 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29581e57e0 con 0x7f29580b3470 2024-01-15T11:26:39.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.063+0000 7f2960564700 1 -- 172.21.15.113:0/3445874801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f29581e57e0 con 0x7f29580b3470 2024-01-15T11:26:39.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.066+0000 7f294effd700 1 -- 172.21.15.113:0/3445874801 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v11)=0 v11) v1 ==== 135+0+0 (secure 0 0 0) 0x7f29581e57e0 con 0x7f29580b3470 2024-01-15T11:26:39.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.068+0000 7f2960564700 1 -- 172.21.15.113:0/3445874801 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f293c06b3e0 msgr2=0x7f293c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.068+0000 7f2960564700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f293c06b3e0 0x7f293c06d8a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2954005a70 tx=0x7f2954001c00 comp rx=0 tx=0).stop 2024-01-15T11:26:39.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.068+0000 7f2960564700 1 -- 172.21.15.113:0/3445874801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29580b3470 msgr2=0x7f29581e5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.068+0000 7f2960564700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29580b3470 0x7f29581e5ac0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f29580d0a20 tx=0x7f2948004750 comp rx=0 tx=0).stop 2024-01-15T11:26:39.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.068+0000 7f2960564700 1 -- 172.21.15.113:0/3445874801 shutdown_connections 2024-01-15T11:26:39.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.068+0000 7f2960564700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f293c06b3e0 0x7f293c06d8a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.068+0000 7f2960564700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f295815df60 0x7f2958146840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.068+0000 7f2960564700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29580b3470 0x7f29581e5ac0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.068+0000 7f2960564700 1 --2- 172.21.15.113:0/3445874801 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2958151620 0x7f2958146300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.068+0000 7f2960564700 1 -- 172.21.15.113:0/3445874801 >> 172.21.15.113:0/3445874801 conn(0x7f295806b410 msgr2=0x7f295806dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:39.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.069+0000 7f2960564700 1 -- 172.21.15.113:0/3445874801 shutdown_connections 2024-01-15T11:26:39.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.069+0000 7f2960564700 1 -- 172.21.15.113:0/3445874801 wait complete. 2024-01-15T11:26:39.081 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T11:26:39.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.209+0000 7ff986c7f700 1 Processor -- start 2024-01-15T11:26:39.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.210+0000 7ff986c7f700 1 -- start start 2024-01-15T11:26:39.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.210+0000 7ff986c7f700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7ff9800b5b60 0x7ff98006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.210+0000 7ff986c7f700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9801624c0 con 0x7ff98015e0b0 2024-01-15T11:26:39.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.210+0000 7ff986c7f700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff980161b90 con 0x7ff98006c710 2024-01-15T11:26:39.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.210+0000 7ff986c7f700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98015e490 con 0x7ff9800b5b60 2024-01-15T11:26:39.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.210+0000 7ff984a1b700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ff98015e0b0 0x7ff98006c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:33728/0 (socket says 172.21.15.113:33728) 2024-01-15T11:26:39.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.210+0000 7ff984a1b700 1 -- 172.21.15.113:0/2894453497 learned_addr learned my addr 172.21.15.113:0/2894453497 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:39.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff98521c700 1 --2- 172.21.15.113:0/2894453497 >> v2:172.21.15.141:3300/0 conn(0x7ff9800b5b60 0x7ff98006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466152022 0 0) 0x7ff9801624c0 con 0x7ff98015e0b0 2024-01-15T11:26:39.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9700035b0 con 0x7ff98015e0b0 2024-01-15T11:26:39.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4008234638 0 0) 0x7ff9700035b0 con 0x7ff98015e0b0 2024-01-15T11:26:39.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 >> v2:172.21.15.141:3300/0 conn(0x7ff9800b5b60 msgr2=0x7ff98006e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:39.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 --2- 172.21.15.113:0/2894453497 >> v2:172.21.15.141:3300/0 conn(0x7ff9800b5b60 0x7ff98006e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 >> v1:172.21.15.141:6789/0 conn(0x7ff98006c710 legacy=0x7ff98006dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff980161e70 con 0x7ff98015e0b0 2024-01-15T11:26:39.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7ff974001670 con 0x7ff98015e0b0 2024-01-15T11:26:39.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 >> v1:172.21.15.113:6789/0 conn(0x7ff98015e0b0 legacy=0x7ff98006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 --2- 172.21.15.113:0/2894453497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff970003a00 0x7ff970003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 --2- 172.21.15.113:0/2894453497 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff970004850 0x7ff970004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 --2- 172.21.15.113:0/2894453497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff970005210 0x7ff97000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98015e490 con 0x7ff970005210 2024-01-15T11:26:39.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97000c1b0 con 0x7ff970003a00 2024-01-15T11:26:39.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff97000c370 con 0x7ff970004850 2024-01-15T11:26:39.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff98521c700 1 --2- 172.21.15.113:0/2894453497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff970005210 0x7ff97000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff984a1b700 1 --2- 172.21.15.113:0/2894453497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff970003a00 0x7ff970003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff97ffff700 1 --2- 172.21.15.113:0/2894453497 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff970004850 0x7ff970004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff98521c700 1 -- 172.21.15.113:0/2894453497 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff970004850 msgr2=0x7ff970004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:39.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff98521c700 1 --2- 172.21.15.113:0/2894453497 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff970004850 0x7ff970004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff98521c700 1 -- 172.21.15.113:0/2894453497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff970003a00 msgr2=0x7ff970003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:39.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff98521c700 1 --2- 172.21.15.113:0/2894453497 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff970003a00 0x7ff970003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff98521c700 1 -- 172.21.15.113:0/2894453497 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9700035b0 con 0x7ff970005210 2024-01-15T11:26:39.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.211+0000 7ff98521c700 1 -- 172.21.15.113:0/2894453497 reap_dead start 2024-01-15T11:26:39.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.212+0000 7ff98521c700 1 --2- 172.21.15.113:0/2894453497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff970005210 0x7ff97000ba10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff978009f90 tx=0x7ff970005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d5af91ac9c5950a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:39.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.212+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978003a20 con 0x7ff970005210 2024-01-15T11:26:39.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.212+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97800f040 con 0x7ff970005210 2024-01-15T11:26:39.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.212+0000 7ff97f7fe700 1 -- 172.21.15.113:0/2894453497 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978003de0 con 0x7ff970005210 2024-01-15T11:26:39.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.212+0000 7ff986c7f700 1 -- 172.21.15.113:0/2894453497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff970005210 msgr2=0x7ff97000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.212+0000 7ff986c7f700 1 --2- 172.21.15.113:0/2894453497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff970005210 0x7ff97000ba10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff978009f90 tx=0x7ff970005880 comp rx=0 tx=0).stop 2024-01-15T11:26:39.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.212+0000 7ff986c7f700 1 -- 172.21.15.113:0/2894453497 shutdown_connections 2024-01-15T11:26:39.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.212+0000 7ff986c7f700 1 --2- 172.21.15.113:0/2894453497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff970005210 0x7ff97000ba10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.212+0000 7ff986c7f700 1 -- 172.21.15.113:0/2894453497 >> 172.21.15.113:0/2894453497 conn(0x7ff980071b50 msgr2=0x7ff980071f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:39.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.212+0000 7ff986c7f700 1 -- 172.21.15.113:0/2894453497 shutdown_connections 2024-01-15T11:26:39.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.212+0000 7ff986c7f700 1 -- 172.21.15.113:0/2894453497 wait complete. 2024-01-15T11:26:39.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.214+0000 7ff986c7f700 1 Processor -- start 2024-01-15T11:26:39.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.214+0000 7ff986c7f700 1 -- start start 2024-01-15T11:26:39.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.214+0000 7ff986c7f700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff970005210 0x7ff98014e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.214+0000 7ff986c7f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff980153380 0x7ff98015d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.214+0000 7ff986c7f700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff98015d5d0 0x7ff9800b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.214+0000 7ff986c7f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98015e5d0 con 0x7ff980153380 2024-01-15T11:26:39.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.214+0000 7ff986c7f700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98014f040 con 0x7ff98015d5d0 2024-01-15T11:26:39.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.214+0000 7ff986c7f700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98015e820 con 0x7ff970005210 2024-01-15T11:26:39.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff97ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff980153380 0x7ff98015d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff97ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff980153380 0x7ff98015d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54048/0 (socket says 172.21.15.113:54048) 2024-01-15T11:26:39.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff97ffff700 1 -- 172.21.15.113:0/1290103236 learned_addr learned my addr 172.21.15.113:0/1290103236 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:39.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff98521c700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff98015d5d0 0x7ff9800b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff984a1b700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff970005210 0x7ff98014e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff97ffff700 1 -- 172.21.15.113:0/1290103236 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff970005210 msgr2=0x7ff98014e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff97ffff700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff970005210 0x7ff98014e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff97ffff700 1 -- 172.21.15.113:0/1290103236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff98015d5d0 msgr2=0x7ff9800b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff97ffff700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff98015d5d0 0x7ff9800b38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff97ffff700 1 -- 172.21.15.113:0/1290103236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98015eb00 con 0x7ff980153380 2024-01-15T11:26:39.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff97ffff700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff980153380 0x7ff98015d090 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00f640 tx=0x7ff96c00f950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:39.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff97d7fa700 1 -- 172.21.15.113:0/1290103236 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0052e0 con 0x7ff980153380 2024-01-15T11:26:39.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff97d7fa700 1 -- 172.21.15.113:0/1290103236 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96c005440 con 0x7ff980153380 2024-01-15T11:26:39.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff986c7f700 1 -- 172.21.15.113:0/1290103236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98015ede0 con 0x7ff980153380 2024-01-15T11:26:39.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff97d7fa700 1 -- 172.21.15.113:0/1290103236 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0038a0 con 0x7ff980153380 2024-01-15T11:26:39.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.215+0000 7ff986c7f700 1 -- 172.21.15.113:0/1290103236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98015f3a0 con 0x7ff980153380 2024-01-15T11:26:39.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.216+0000 7ff986c7f700 1 -- 172.21.15.113:0/1290103236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98015f560 con 0x7ff980153380 2024-01-15T11:26:39.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.217+0000 7ff97d7fa700 1 -- 172.21.15.113:0/1290103236 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7ff96c003a00 con 0x7ff980153380 2024-01-15T11:26:39.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.217+0000 7ff97d7fa700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff96406b3e0 0x7ff96406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.217+0000 7ff97d7fa700 1 -- 172.21.15.113:0/1290103236 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff96c089720 con 0x7ff980153380 2024-01-15T11:26:39.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.217+0000 7ff984a1b700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff96406b3e0 0x7ff96406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.217+0000 7ff984a1b700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff96406b3e0 0x7ff96406d8a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff980152880 tx=0x7ff974000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:39.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.221+0000 7ff97d7fa700 1 -- 172.21.15.113:0/1290103236 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff96c0545a0 con 0x7ff980153380 2024-01-15T11:26:39.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.452+0000 7ff986c7f700 1 -- 172.21.15.113:0/1290103236 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9800b6040 con 0x7ff96406b3e0 2024-01-15T11:26:39.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.465+0000 7ff97d7fa700 1 -- 172.21.15.113:0/1290103236 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff974007e80 con 0x7ff96406b3e0 2024-01-15T11:26:39.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.467+0000 7ff986c7f700 1 -- 172.21.15.113:0/1290103236 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff96406b3e0 msgr2=0x7ff96406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.467+0000 7ff986c7f700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff96406b3e0 0x7ff96406d8a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff980152880 tx=0x7ff974000ed0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.467+0000 7ff986c7f700 1 -- 172.21.15.113:0/1290103236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff980153380 msgr2=0x7ff98015d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.467+0000 7ff986c7f700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff980153380 0x7ff98015d090 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00f640 tx=0x7ff96c00f950 comp rx=0 tx=0).stop 2024-01-15T11:26:39.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.467+0000 7ff986c7f700 1 -- 172.21.15.113:0/1290103236 shutdown_connections 2024-01-15T11:26:39.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.467+0000 7ff986c7f700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ff96406b3e0 0x7ff96406d8a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.467+0000 7ff986c7f700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff98015d5d0 0x7ff9800b38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.467+0000 7ff986c7f700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ff970005210 0x7ff98014e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.467+0000 7ff986c7f700 1 --2- 172.21.15.113:0/1290103236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff980153380 0x7ff98015d090 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.467+0000 7ff986c7f700 1 -- 172.21.15.113:0/1290103236 >> 172.21.15.113:0/1290103236 conn(0x7ff980071b50 msgr2=0x7ff98006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:39.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.467+0000 7ff986c7f700 1 -- 172.21.15.113:0/1290103236 shutdown_connections 2024-01-15T11:26:39.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.467+0000 7ff986c7f700 1 -- 172.21.15.113:0/1290103236 wait complete. 2024-01-15T11:26:39.480 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T11:26:39.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.605+0000 7f26f2b39700 1 Processor -- start 2024-01-15T11:26:39.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.605+0000 7f26f2b39700 1 -- start start 2024-01-15T11:26:39.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.606+0000 7f26f2b39700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f26ec06f580 0x7f26ec15e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.606+0000 7f26f2b39700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f26ec15ec50 0x7f26ec06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.606+0000 7f26f2b39700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec0b5600 con 0x7f26ec158600 2024-01-15T11:26:39.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.606+0000 7f26f2b39700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26ec15f0e0 con 0x7f26ec15ec50 2024-01-15T11:26:39.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.606+0000 7f26f2b39700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec06fa10 con 0x7f26ec06f580 2024-01-15T11:26:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.606+0000 7f26f10d6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f26ec15ec50 0x7f26ec06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.606+0000 7f26f10d6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f26ec15ec50 0x7f26ec06aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54056/0 (socket says 172.21.15.113:54056) 2024-01-15T11:26:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.606+0000 7f26f10d6700 1 -- 172.21.15.113:0/3450605664 learned_addr learned my addr 172.21.15.113:0/3450605664 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.607+0000 7f26ebfff700 1 --2- 172.21.15.113:0/3450605664 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f26ec06f580 0x7f26ec15e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.607+0000 7f26f10d6700 1 -- 172.21.15.113:0/3450605664 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f26ec06f580 msgr2=0x7f26ec15e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.607+0000 7f26f10d6700 1 --2- 172.21.15.113:0/3450605664 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f26ec06f580 0x7f26ec15e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.607+0000 7f26f10d6700 1 -- 172.21.15.113:0/3450605664 >> v1:172.21.15.113:6789/0 conn(0x7f26ec158600 legacy=0x7f26ec0b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.607+0000 7f26f10d6700 1 -- 172.21.15.113:0/3450605664 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec06e4e0 con 0x7f26ec15ec50 2024-01-15T11:26:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.607+0000 7f26f10d6700 1 --2- 172.21.15.113:0/3450605664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26ec15ec50 0x7f26ec06aff0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009e50 tx=0x7f26e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74236a63b1d76bd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.607+0000 7f26eb7fe700 1 -- 172.21.15.113:0/3450605664 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e400e070 con 0x7f26ec15ec50 2024-01-15T11:26:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.607+0000 7f26eb7fe700 1 -- 172.21.15.113:0/3450605664 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e400b9c0 con 0x7f26ec15ec50 2024-01-15T11:26:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.607+0000 7f26eb7fe700 1 -- 172.21.15.113:0/3450605664 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e400bd70 con 0x7f26ec15ec50 2024-01-15T11:26:39.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.607+0000 7f26f2b39700 1 -- 172.21.15.113:0/3450605664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26ec15ec50 msgr2=0x7f26ec06aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.607+0000 7f26f2b39700 1 --2- 172.21.15.113:0/3450605664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26ec15ec50 0x7f26ec06aff0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009e50 tx=0x7f26e400b330 comp rx=0 tx=0).stop 2024-01-15T11:26:39.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.608+0000 7f26f2b39700 1 -- 172.21.15.113:0/3450605664 shutdown_connections 2024-01-15T11:26:39.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.608+0000 7f26f2b39700 1 --2- 172.21.15.113:0/3450605664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26ec15ec50 0x7f26ec06aff0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.608+0000 7f26f2b39700 1 --2- 172.21.15.113:0/3450605664 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f26ec06f580 0x7f26ec15e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.608+0000 7f26f2b39700 1 -- 172.21.15.113:0/3450605664 >> 172.21.15.113:0/3450605664 conn(0x7f26ec0b3250 msgr2=0x7f26ec0d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:39.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.608+0000 7f26f2b39700 1 -- 172.21.15.113:0/3450605664 shutdown_connections 2024-01-15T11:26:39.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.608+0000 7f26f2b39700 1 -- 172.21.15.113:0/3450605664 wait complete. 2024-01-15T11:26:39.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.609+0000 7f26f2b39700 1 Processor -- start 2024-01-15T11:26:39.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.609+0000 7f26f2b39700 1 -- start start 2024-01-15T11:26:39.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f2b39700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26ec06f580 0x7f26ec146710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f2b39700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26ec158600 0x7f26ec15e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f2b39700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f26ec15ec50 0x7f26ec15c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f2b39700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec146c50 con 0x7f26ec06f580 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f2b39700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec15d250 con 0x7f26ec158600 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f2b39700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec15d530 con 0x7f26ec15ec50 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f08d5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26ec06f580 0x7f26ec146710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f08d5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26ec06f580 0x7f26ec146710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54072/0 (socket says 172.21.15.113:54072) 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f08d5700 1 -- 172.21.15.113:0/960918838 learned_addr learned my addr 172.21.15.113:0/960918838 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26ebfff700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26ec158600 0x7f26ec15e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f10d6700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f26ec15ec50 0x7f26ec15c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f08d5700 1 -- 172.21.15.113:0/960918838 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f26ec15ec50 msgr2=0x7f26ec15c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f08d5700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f26ec15ec50 0x7f26ec15c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f08d5700 1 -- 172.21.15.113:0/960918838 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26ec158600 msgr2=0x7f26ec15e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f08d5700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26ec158600 0x7f26ec15e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f08d5700 1 -- 172.21.15.113:0/960918838 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e4009b30 con 0x7f26ec06f580 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26ebfff700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26ec158600 0x7f26ec15e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.610+0000 7f26f10d6700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f26ec15ec50 0x7f26ec15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.611+0000 7f26f08d5700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26ec06f580 0x7f26ec146710 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f26e000c4e0 tx=0x7f26e000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.611+0000 7f26e97fa700 1 -- 172.21.15.113:0/960918838 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00052d0 con 0x7f26ec06f580 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.611+0000 7f26e97fa700 1 -- 172.21.15.113:0/960918838 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e0005430 con 0x7f26ec06f580 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.611+0000 7f26f2b39700 1 -- 172.21.15.113:0/960918838 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ec15d810 con 0x7f26ec06f580 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.611+0000 7f26e97fa700 1 -- 172.21.15.113:0/960918838 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00038c0 con 0x7f26ec06f580 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.611+0000 7f26f2b39700 1 -- 172.21.15.113:0/960918838 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ec15ddd0 con 0x7f26ec06f580 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.612+0000 7f26f2b39700 1 -- 172.21.15.113:0/960918838 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26ec15dff0 con 0x7f26ec06f580 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.612+0000 7f26e97fa700 1 -- 172.21.15.113:0/960918838 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f26e0003a20 con 0x7f26ec06f580 2024-01-15T11:26:39.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.612+0000 7f26e97fa700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f26d006b410 0x7f26d006d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:39.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.613+0000 7f26e97fa700 1 -- 172.21.15.113:0/960918838 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26e0089120 con 0x7f26ec06f580 2024-01-15T11:26:39.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.613+0000 7f26ebfff700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f26d006b410 0x7f26d006d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:39.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.613+0000 7f26ebfff700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f26d006b410 0x7f26d006d8d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f26ec074c40 tx=0x7f26d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:39.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.617+0000 7f26e97fa700 1 -- 172.21.15.113:0/960918838 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26e0053f10 con 0x7f26ec06f580 2024-01-15T11:26:39.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.839+0000 7f26f2b39700 1 -- 172.21.15.113:0/960918838 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f26ec06ed80 con 0x7f26d006b410 2024-01-15T11:26:39.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.845+0000 7f26e97fa700 1 -- 172.21.15.113:0/960918838 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f26d8009610 con 0x7f26d006b410 2024-01-15T11:26:39.844 INFO:teuthology.orchestra.run.smithi113.stdout:/volumes/_nogroup/sv1/495e2b63-0db2-4f7a-b1b1-d2919dd37cc5 2024-01-15T11:26:39.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.847+0000 7f26f2b39700 1 -- 172.21.15.113:0/960918838 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f26d006b410 msgr2=0x7f26d006d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.847+0000 7f26f2b39700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f26d006b410 0x7f26d006d8d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f26ec074c40 tx=0x7f26d8008040 comp rx=0 tx=0).stop 2024-01-15T11:26:39.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.847+0000 7f26f2b39700 1 -- 172.21.15.113:0/960918838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26ec06f580 msgr2=0x7f26ec146710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:39.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.847+0000 7f26f2b39700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26ec06f580 0x7f26ec146710 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f26e000c4e0 tx=0x7f26e000c8a0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.847+0000 7f26f2b39700 1 -- 172.21.15.113:0/960918838 shutdown_connections 2024-01-15T11:26:39.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.847+0000 7f26f2b39700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f26d006b410 0x7f26d006d8d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.847+0000 7f26f2b39700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f26ec15ec50 0x7f26ec15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.847+0000 7f26f2b39700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26ec06f580 0x7f26ec146710 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.847+0000 7f26f2b39700 1 --2- 172.21.15.113:0/960918838 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f26ec158600 0x7f26ec15e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:39.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.847+0000 7f26f2b39700 1 -- 172.21.15.113:0/960918838 >> 172.21.15.113:0/960918838 conn(0x7f26ec0b3250 msgr2=0x7f26ec158e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:39.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.847+0000 7f26f2b39700 1 -- 172.21.15.113:0/960918838 shutdown_connections 2024-01-15T11:26:39.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:39.847+0000 7f26f2b39700 1 -- 172.21.15.113:0/960918838 wait complete. 2024-01-15T11:26:39.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:39.858 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/495e2b63-0db2-4f7a-b1b1-d2919dd37cc5) 2024-01-15T11:26:39.876 INFO:teuthology.orchestra.run.smithi113.stdout:# file: volumes/_nogroup/sv1/495e2b63-0db2-4f7a-b1b1-d2919dd37cc5 2024-01-15T11:26:39.876 INFO:teuthology.orchestra.run.smithi113.stdout:ceph.dir.rbytes="2972172288" 2024-01-15T11:26:39.876 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:39.876 INFO:tasks.cephfs.test_volumes:r_size = 2972172288 2024-01-15T11:26:41.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:41.878 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/495e2b63-0db2-4f7a-b1b1-d2919dd37cc5) 2024-01-15T11:26:41.900 INFO:teuthology.orchestra.run.smithi113.stdout:# file: volumes/_nogroup/sv1/495e2b63-0db2-4f7a-b1b1-d2919dd37cc5 2024-01-15T11:26:41.900 INFO:teuthology.orchestra.run.smithi113.stdout:ceph.dir.rbytes="2972172288" 2024-01-15T11:26:41.901 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:41.901 INFO:tasks.cephfs.test_volumes:r_size = 2972172288 2024-01-15T11:26:43.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:26:43.902 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/495e2b63-0db2-4f7a-b1b1-d2919dd37cc5) 2024-01-15T11:26:43.924 INFO:teuthology.orchestra.run.smithi113.stdout:# file: volumes/_nogroup/sv1/495e2b63-0db2-4f7a-b1b1-d2919dd37cc5 2024-01-15T11:26:43.925 INFO:teuthology.orchestra.run.smithi113.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T11:26:43.925 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:43.925 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T11:26:43.925 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-15T11:26:44.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.105+0000 7f006c419700 1 Processor -- start 2024-01-15T11:26:44.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.105+0000 7f006c419700 1 -- start start 2024-01-15T11:26:44.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.106+0000 7f006c419700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f006406c100 0x7f0064074710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.106+0000 7f006c419700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0064158950 con 0x7f006406ea30 2024-01-15T11:26:44.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.106+0000 7f006c419700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006415e670 con 0x7f0064158570 2024-01-15T11:26:44.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.106+0000 7f006c419700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006415f2f0 con 0x7f006406c100 2024-01-15T11:26:44.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.106+0000 7f00699b4700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f006406ea30 0x7f0064074000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:33740/0 (socket says 172.21.15.113:33740) 2024-01-15T11:26:44.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.106+0000 7f00699b4700 1 -- 172.21.15.113:0/2205425575 learned_addr learned my addr 172.21.15.113:0/2205425575 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:44.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.106+0000 7f006a9b6700 1 --2- 172.21.15.113:0/2205425575 >> v2:172.21.15.141:3300/0 conn(0x7f006406c100 0x7f0064074710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.106+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538126454 0 0) 0x7f0064158950 con 0x7f006406ea30 2024-01-15T11:26:44.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.106+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00500035b0 con 0x7f006406ea30 2024-01-15T11:26:44.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3033749337 0 0) 0x7f00500035b0 con 0x7f006406ea30 2024-01-15T11:26:44.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 >> v2:172.21.15.141:3300/0 conn(0x7f006406c100 msgr2=0x7f0064074710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:44.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 --2- 172.21.15.113:0/2205425575 >> v2:172.21.15.141:3300/0 conn(0x7f006406c100 0x7f0064074710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 >> v1:172.21.15.141:6789/0 conn(0x7f0064158570 legacy=0x7f006406e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:44.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006415e950 con 0x7f006406ea30 2024-01-15T11:26:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f0060001670 con 0x7f006406ea30 2024-01-15T11:26:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 >> v1:172.21.15.113:6789/0 conn(0x7f006406ea30 legacy=0x7f0064074000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 --2- 172.21.15.113:0/2205425575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0050003a00 0x7f0050003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 --2- 172.21.15.113:0/2205425575 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0050004850 0x7f0050004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 --2- 172.21.15.113:0/2205425575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0050005210 0x7f005000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006415f2f0 con 0x7f0050005210 2024-01-15T11:26:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005000c0b0 con 0x7f0050003a00 2024-01-15T11:26:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005000c270 con 0x7f0050004850 2024-01-15T11:26:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f006a9b6700 1 --2- 172.21.15.113:0/2205425575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0050005210 0x7f005000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f006a1b5700 1 --2- 172.21.15.113:0/2205425575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0050003a00 0x7f0050003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00699b4700 1 --2- 172.21.15.113:0/2205425575 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0050004850 0x7f0050004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f006a9b6700 1 -- 172.21.15.113:0/2205425575 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0050004850 msgr2=0x7f0050004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f006a9b6700 1 --2- 172.21.15.113:0/2205425575 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0050004850 0x7f0050004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f006a9b6700 1 -- 172.21.15.113:0/2205425575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0050003a00 msgr2=0x7f0050003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f006a9b6700 1 --2- 172.21.15.113:0/2205425575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0050003a00 0x7f0050003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f006a9b6700 1 -- 172.21.15.113:0/2205425575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00500035b0 con 0x7f0050005210 2024-01-15T11:26:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00699b4700 1 --2- 172.21.15.113:0/2205425575 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0050004850 0x7f0050004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f006a9b6700 1 -- 172.21.15.113:0/2205425575 reap_dead start 2024-01-15T11:26:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f006a9b6700 1 --2- 172.21.15.113:0/2205425575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0050005210 0x7f005000ba10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f005c009f90 tx=0x7f0050005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4f1320fba61a43 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c003a20 con 0x7f0050005210 2024-01-15T11:26:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c00f040 con 0x7f0050005210 2024-01-15T11:26:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.107+0000 7f00691b3700 1 -- 172.21.15.113:0/2205425575 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c003de0 con 0x7f0050005210 2024-01-15T11:26:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.108+0000 7f006c419700 1 -- 172.21.15.113:0/2205425575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0050005210 msgr2=0x7f005000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.108+0000 7f006c419700 1 --2- 172.21.15.113:0/2205425575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0050005210 0x7f005000ba10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f005c009f90 tx=0x7f0050005880 comp rx=0 tx=0).stop 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.108+0000 7f006c419700 1 -- 172.21.15.113:0/2205425575 shutdown_connections 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.108+0000 7f006c419700 1 --2- 172.21.15.113:0/2205425575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0050005210 0x7f005000ba10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.108+0000 7f006c419700 1 -- 172.21.15.113:0/2205425575 >> 172.21.15.113:0/2205425575 conn(0x7f00640cd6b0 msgr2=0x7f00640cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.108+0000 7f006c419700 1 -- 172.21.15.113:0/2205425575 shutdown_connections 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.108+0000 7f006c419700 1 -- 172.21.15.113:0/2205425575 wait complete. 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.109+0000 7f006c419700 1 Processor -- start 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.109+0000 7f006c419700 1 -- start start 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006c419700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0050005210 0x7f006414f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006c419700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0064151eb0 0x7f00641f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006c419700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0064152330 0x7f00641f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006c419700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0064151a20 con 0x7f0064152330 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006c419700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0064150e10 con 0x7f0064151eb0 2024-01-15T11:26:44.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006c419700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00641510f0 con 0x7f0050005210 2024-01-15T11:26:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006a9b6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0064152330 0x7f00641f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006a9b6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0064152330 0x7f00641f4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54092/0 (socket says 172.21.15.113:54092) 2024-01-15T11:26:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006a9b6700 1 -- 172.21.15.113:0/3476750690 learned_addr learned my addr 172.21.15.113:0/3476750690 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f00699b4700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0064151eb0 0x7f00641f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006a1b5700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0050005210 0x7f006414f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006a9b6700 1 -- 172.21.15.113:0/3476750690 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0050005210 msgr2=0x7f006414f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006a9b6700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0050005210 0x7f006414f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006a9b6700 1 -- 172.21.15.113:0/3476750690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0064151eb0 msgr2=0x7f00641f3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006a9b6700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0064151eb0 0x7f00641f3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f006a9b6700 1 -- 172.21.15.113:0/3476750690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00641513d0 con 0x7f0064152330 2024-01-15T11:26:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.110+0000 7f00699b4700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0064151eb0 0x7f00641f3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.111+0000 7f006a9b6700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0064152330 0x7f00641f4190 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f006406fb60 tx=0x7f005c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.111+0000 7f005affd700 1 -- 172.21.15.113:0/3476750690 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c003e10 con 0x7f0064152330 2024-01-15T11:26:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.111+0000 7f005affd700 1 -- 172.21.15.113:0/3476750690 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c004230 con 0x7f0064152330 2024-01-15T11:26:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.111+0000 7f006c419700 1 -- 172.21.15.113:0/3476750690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00641516b0 con 0x7f0064152330 2024-01-15T11:26:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.111+0000 7f005affd700 1 -- 172.21.15.113:0/3476750690 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c0045e0 con 0x7f0064152330 2024-01-15T11:26:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.111+0000 7f006c419700 1 -- 172.21.15.113:0/3476750690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006414f750 con 0x7f0064152330 2024-01-15T11:26:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.112+0000 7f006c419700 1 -- 172.21.15.113:0/3476750690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0064151b90 con 0x7f0064152330 2024-01-15T11:26:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.112+0000 7f005affd700 1 -- 172.21.15.113:0/3476750690 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f005c004740 con 0x7f0064152330 2024-01-15T11:26:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.112+0000 7f005affd700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f004806b3e0 0x7f004806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.113+0000 7f005affd700 1 -- 172.21.15.113:0/3476750690 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f005c08adb0 con 0x7f0064152330 2024-01-15T11:26:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.113+0000 7f006a1b5700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f004806b3e0 0x7f004806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.113+0000 7f006a1b5700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f004806b3e0 0x7f004806d8a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f00641609b0 tx=0x7f0054009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.117+0000 7f005affd700 1 -- 172.21.15.113:0/3476750690 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f005c055cd0 con 0x7f0064152330 2024-01-15T11:26:44.271 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:26:44.274 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:26:44.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.350+0000 7f006c419700 1 -- 172.21.15.113:0/3476750690 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f006415ef10 con 0x7f004806b3e0 2024-01-15T11:26:44.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.404+0000 7f005affd700 1 -- 172.21.15.113:0/3476750690 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0054005290 con 0x7f004806b3e0 2024-01-15T11:26:44.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.407+0000 7f006c419700 1 -- 172.21.15.113:0/3476750690 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f004806b3e0 msgr2=0x7f004806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:44.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.407+0000 7f006c419700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f004806b3e0 0x7f004806d8a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f00641609b0 tx=0x7f0054009040 comp rx=0 tx=0).stop 2024-01-15T11:26:44.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.407+0000 7f006c419700 1 -- 172.21.15.113:0/3476750690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0064152330 msgr2=0x7f00641f4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:44.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.407+0000 7f006c419700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0064152330 0x7f00641f4190 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f006406fb60 tx=0x7f005c004b40 comp rx=0 tx=0).stop 2024-01-15T11:26:44.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.407+0000 7f006c419700 1 -- 172.21.15.113:0/3476750690 shutdown_connections 2024-01-15T11:26:44.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.407+0000 7f006c419700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f004806b3e0 0x7f004806d8a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.407+0000 7f006c419700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0064152330 0x7f00641f4190 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.407+0000 7f006c419700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0050005210 0x7f006414f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.407+0000 7f006c419700 1 --2- 172.21.15.113:0/3476750690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0064151eb0 0x7f00641f3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.407+0000 7f006c419700 1 -- 172.21.15.113:0/3476750690 >> 172.21.15.113:0/3476750690 conn(0x7f00640cd6b0 msgr2=0x7f00640cce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:44.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.407+0000 7f006c419700 1 -- 172.21.15.113:0/3476750690 shutdown_connections 2024-01-15T11:26:44.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.408+0000 7f006c419700 1 -- 172.21.15.113:0/3476750690 wait complete. 2024-01-15T11:26:44.408 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:44.408+0000 7fa4cecb2700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:44.408 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:44.408+0000 7fa4cecb2700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:44.408 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:44.408+0000 7fa4cecb2700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:44.408 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:44.408+0000 7fa4cecb2700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:44.409 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:44.408+0000 7fa4cecb2700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:44.421 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-15T11:26:44.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T11:26:44.422 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:44.422+0000 7fa4ce4b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:44.422 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:44.422+0000 7fa4ce4b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:44.422 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:44.422+0000 7fa4ce4b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:44.422 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:44.422+0000 7fa4ce4b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:44.422 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:26:44.422+0000 7fa4ce4b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:26:44.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.587+0000 7f2d36eb2700 1 Processor -- start 2024-01-15T11:26:44.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.587+0000 7f2d36eb2700 1 -- start start 2024-01-15T11:26:44.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.588+0000 7f2d36eb2700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2d3015e0c0 0x7f2d300afb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.588+0000 7f2d36eb2700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2d300b3bf0 0x7f2d30073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.588+0000 7f2d36eb2700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3015e4a0 con 0x7f2d300744d0 2024-01-15T11:26:44.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.588+0000 7f2d36eb2700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d30074960 con 0x7f2d3015e0c0 2024-01-15T11:26:44.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.588+0000 7f2d36eb2700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3006aff0 con 0x7f2d300b3bf0 2024-01-15T11:26:44.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.588+0000 7f2d2ffff700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2d300b3bf0 0x7f2d30073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.588+0000 7f2d34c4e700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2d3015e0c0 0x7f2d300afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.588+0000 7f2d2ffff700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2d300b3bf0 0x7f2d30073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:55862/0 (socket says 172.21.15.113:55862) 2024-01-15T11:26:44.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.588+0000 7f2d34c4e700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f2d3015e0c0 0x7f2d300afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:36550/0 (socket says 172.21.15.113:36550) 2024-01-15T11:26:44.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.588+0000 7f2d2ffff700 1 -- 172.21.15.113:0/3603288424 learned_addr learned my addr 172.21.15.113:0/3603288424 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:44.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.589+0000 7f2d2f7fe700 1 -- 172.21.15.113:0/3603288424 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190201375 0 0) 0x7f2d3015e4a0 con 0x7f2d300744d0 2024-01-15T11:26:44.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.589+0000 7f2d2f7fe700 1 -- 172.21.15.113:0/3603288424 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d200035b0 con 0x7f2d300744d0 2024-01-15T11:26:44.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.589+0000 7f2d34c4e700 1 -- 172.21.15.113:0/3603288424 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2d300b3bf0 msgr2=0x7f2d30073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:44.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.589+0000 7f2d34c4e700 1 --2- 172.21.15.113:0/3603288424 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2d300b3bf0 0x7f2d30073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.589+0000 7f2d34c4e700 1 -- 172.21.15.113:0/3603288424 >> v1:172.21.15.141:6789/0 conn(0x7f2d300744d0 legacy=0x7f2d30161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:44.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.589+0000 7f2d34c4e700 1 -- 172.21.15.113:0/3603288424 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3006b2d0 con 0x7f2d3015e0c0 2024-01-15T11:26:44.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.589+0000 7f2d2ffff700 1 --2- 172.21.15.113:0/3603288424 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2d300b3bf0 0x7f2d30073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:26:44.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.589+0000 7f2d34c4e700 1 --2- 172.21.15.113:0/3603288424 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d3015e0c0 0x7f2d300afb80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009840 tx=0x7f2d24009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eaed2aad456c78e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:44.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.589+0000 7f2d2f7fe700 1 -- 172.21.15.113:0/3603288424 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24011830 con 0x7f2d3015e0c0 2024-01-15T11:26:44.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.589+0000 7f2d2f7fe700 1 -- 172.21.15.113:0/3603288424 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d24011990 con 0x7f2d3015e0c0 2024-01-15T11:26:44.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.589+0000 7f2d2f7fe700 1 -- 172.21.15.113:0/3603288424 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24011d20 con 0x7f2d3015e0c0 2024-01-15T11:26:44.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.590+0000 7f2d36eb2700 1 -- 172.21.15.113:0/3603288424 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d3015e0c0 msgr2=0x7f2d300afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:44.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.590+0000 7f2d36eb2700 1 --2- 172.21.15.113:0/3603288424 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d3015e0c0 0x7f2d300afb80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009840 tx=0x7f2d24009b50 comp rx=0 tx=0).stop 2024-01-15T11:26:44.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.590+0000 7f2d36eb2700 1 -- 172.21.15.113:0/3603288424 shutdown_connections 2024-01-15T11:26:44.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.590+0000 7f2d36eb2700 1 --2- 172.21.15.113:0/3603288424 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d3015e0c0 0x7f2d300afb80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.590+0000 7f2d36eb2700 1 --2- 172.21.15.113:0/3603288424 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2d300b3bf0 0x7f2d30073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.590+0000 7f2d36eb2700 1 -- 172.21.15.113:0/3603288424 >> 172.21.15.113:0/3603288424 conn(0x7f2d300d0c00 msgr2=0x7f2d300d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:44.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.590+0000 7f2d36eb2700 1 -- 172.21.15.113:0/3603288424 shutdown_connections 2024-01-15T11:26:44.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.590+0000 7f2d36eb2700 1 -- 172.21.15.113:0/3603288424 wait complete. 2024-01-15T11:26:44.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.591+0000 7f2d36eb2700 1 Processor -- start 2024-01-15T11:26:44.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.591+0000 7f2d36eb2700 1 -- start start 2024-01-15T11:26:44.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d36eb2700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2d300744d0 0x7f2d3014c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d36eb2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d300b3bf0 0x7f2d300cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d36eb2700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d3015e0c0 0x7f2d301496a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d36eb2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d300ce7b0 con 0x7f2d300b3bf0 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d36eb2700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3015cc40 con 0x7f2d3015e0c0 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d36eb2700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d300ce360 con 0x7f2d300744d0 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d2ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d300b3bf0 0x7f2d300cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d2ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d300b3bf0 0x7f2d300cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36352/0 (socket says 172.21.15.113:36352) 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d2ffff700 1 -- 172.21.15.113:0/2816075906 learned_addr learned my addr 172.21.15.113:0/2816075906 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d34c4e700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2d300744d0 0x7f2d3014c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d3544f700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d3015e0c0 0x7f2d301496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d2ffff700 1 -- 172.21.15.113:0/2816075906 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2d300744d0 msgr2=0x7f2d3014c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d2ffff700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2d300744d0 0x7f2d3014c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d2ffff700 1 -- 172.21.15.113:0/2816075906 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d3015e0c0 msgr2=0x7f2d301496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d2ffff700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d3015e0c0 0x7f2d301496a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.592+0000 7f2d2ffff700 1 -- 172.21.15.113:0/2816075906 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1c009550 con 0x7f2d300b3bf0 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.593+0000 7f2d2ffff700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d300b3bf0 0x7f2d300cde20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00ebe0 tx=0x7f2d1c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.593+0000 7f2d2d7fa700 1 -- 172.21.15.113:0/2816075906 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c00c850 con 0x7f2d300b3bf0 2024-01-15T11:26:44.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.593+0000 7f2d2d7fa700 1 -- 172.21.15.113:0/2816075906 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c00c9b0 con 0x7f2d300b3bf0 2024-01-15T11:26:44.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.593+0000 7f2d36eb2700 1 -- 172.21.15.113:0/2816075906 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d240094a0 con 0x7f2d300b3bf0 2024-01-15T11:26:44.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.593+0000 7f2d2d7fa700 1 -- 172.21.15.113:0/2816075906 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c00cd40 con 0x7f2d300b3bf0 2024-01-15T11:26:44.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.593+0000 7f2d36eb2700 1 -- 172.21.15.113:0/2816075906 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d3015d170 con 0x7f2d300b3bf0 2024-01-15T11:26:44.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.594+0000 7f2d36eb2700 1 -- 172.21.15.113:0/2816075906 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d300ce960 con 0x7f2d300b3bf0 2024-01-15T11:26:44.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.594+0000 7f2d2d7fa700 1 -- 172.21.15.113:0/2816075906 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f2d1c013470 con 0x7f2d300b3bf0 2024-01-15T11:26:44.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.595+0000 7f2d2d7fa700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2d1406b3e0 0x7f2d1406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:44.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.595+0000 7f2d2d7fa700 1 -- 172.21.15.113:0/2816075906 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d1c08a090 con 0x7f2d300b3bf0 2024-01-15T11:26:44.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.595+0000 7f2d34c4e700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2d1406b3e0 0x7f2d1406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:44.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.595+0000 7f2d34c4e700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2d1406b3e0 0x7f2d1406d8a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009fd0 tx=0x7f2d24008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:44.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.599+0000 7f2d2d7fa700 1 -- 172.21.15.113:0/2816075906 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d300ce960 con 0x7f2d300b3bf0 2024-01-15T11:26:44.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:44.833+0000 7f2d36eb2700 1 -- 172.21.15.113:0/2816075906 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d3006bb70 con 0x7f2d1406b3e0 2024-01-15T11:26:45.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:45.330+0000 7f2d2d7fa700 1 -- 172.21.15.113:0/2816075906 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f2d1c0552e0 con 0x7f2d300b3bf0 2024-01-15T11:26:48.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.686+0000 7f2d2d7fa700 1 -- 172.21.15.113:0/2816075906 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f2d24011630 con 0x7f2d1406b3e0 2024-01-15T11:26:48.686 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:26:48.686 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-15T11:26:48.686 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-15T11:26:48.687 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-15T11:26:48.687 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-15T11:26:48.687 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-15T11:26:48.687 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-15T11:26:48.687 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:48.687 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:48.687 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:26:48.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.690+0000 7f2d36eb2700 1 -- 172.21.15.113:0/2816075906 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2d1406b3e0 msgr2=0x7f2d1406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:48.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.690+0000 7f2d36eb2700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2d1406b3e0 0x7f2d1406d8a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009fd0 tx=0x7f2d24008030 comp rx=0 tx=0).stop 2024-01-15T11:26:48.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.690+0000 7f2d36eb2700 1 -- 172.21.15.113:0/2816075906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d300b3bf0 msgr2=0x7f2d300cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:48.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.690+0000 7f2d36eb2700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d300b3bf0 0x7f2d300cde20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00ebe0 tx=0x7f2d1c00eef0 comp rx=0 tx=0).stop 2024-01-15T11:26:48.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.690+0000 7f2d36eb2700 1 -- 172.21.15.113:0/2816075906 shutdown_connections 2024-01-15T11:26:48.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.690+0000 7f2d36eb2700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f2d1406b3e0 0x7f2d1406d8a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:48.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.690+0000 7f2d36eb2700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d3015e0c0 0x7f2d301496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:48.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.690+0000 7f2d36eb2700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f2d300744d0 0x7f2d3014c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:48.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.690+0000 7f2d36eb2700 1 --2- 172.21.15.113:0/2816075906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d300b3bf0 0x7f2d300cde20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:48.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.690+0000 7f2d36eb2700 1 -- 172.21.15.113:0/2816075906 >> 172.21.15.113:0/2816075906 conn(0x7f2d300d0c00 msgr2=0x7f2d300b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:48.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.690+0000 7f2d36eb2700 1 -- 172.21.15.113:0/2816075906 shutdown_connections 2024-01-15T11:26:48.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.690+0000 7f2d36eb2700 1 -- 172.21.15.113:0/2816075906 wait complete. 2024-01-15T11:26:48.705 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T11:26:48.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.831+0000 7fde73100700 1 Processor -- start 2024-01-15T11:26:48.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.832+0000 7fde73100700 1 -- start start 2024-01-15T11:26:48.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.832+0000 7fde73100700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fde6c161c90 0x7fde6c162070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:48.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.832+0000 7fde73100700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fde6c099ba0 0x7fde6c0cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:48.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.832+0000 7fde73100700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c1625b0 con 0x7fde6c15e370 2024-01-15T11:26:48.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.832+0000 7fde73100700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6c0d11b0 con 0x7fde6c099ba0 2024-01-15T11:26:48.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.832+0000 7fde73100700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6c0d05a0 con 0x7fde6c161c90 2024-01-15T11:26:48.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.832+0000 7fde6bfff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fde6c099ba0 0x7fde6c0cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:48.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.832+0000 7fde7169d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fde6c15e370 0x7fde6c15e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:33764/0 (socket says 172.21.15.113:33764) 2024-01-15T11:26:48.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.832+0000 7fde7169d700 1 -- 172.21.15.113:0/2473956865 learned_addr learned my addr 172.21.15.113:0/2473956865 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:48.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.833+0000 7fde70e9c700 1 --2- 172.21.15.113:0/2473956865 >> v2:172.21.15.141:3300/0 conn(0x7fde6c161c90 0x7fde6c162070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:48.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.833+0000 7fde6bfff700 1 -- 172.21.15.113:0/2473956865 >> v2:172.21.15.141:3300/0 conn(0x7fde6c161c90 msgr2=0x7fde6c162070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:48.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.833+0000 7fde6bfff700 1 --2- 172.21.15.113:0/2473956865 >> v2:172.21.15.141:3300/0 conn(0x7fde6c161c90 0x7fde6c162070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:48.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.833+0000 7fde6bfff700 1 -- 172.21.15.113:0/2473956865 >> v1:172.21.15.113:6789/0 conn(0x7fde6c15e370 legacy=0x7fde6c15e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:48.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.833+0000 7fde6bfff700 1 -- 172.21.15.113:0/2473956865 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0d0880 con 0x7fde6c099ba0 2024-01-15T11:26:48.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.833+0000 7fde6b7fe700 1 -- 172.21.15.113:0/2473956865 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100403079 0 0) 0x7fde6c1625b0 con 0x7fde6c15e370 2024-01-15T11:26:48.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.833+0000 7fde6bfff700 1 --2- 172.21.15.113:0/2473956865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde6c099ba0 0x7fde6c0cfce0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0b6450 tx=0x7fde58009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5f2494193c01f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:48.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.833+0000 7fde6b7fe700 1 -- 172.21.15.113:0/2473956865 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58011830 con 0x7fde6c099ba0 2024-01-15T11:26:48.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.833+0000 7fde6b7fe700 1 -- 172.21.15.113:0/2473956865 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde58011990 con 0x7fde6c099ba0 2024-01-15T11:26:48.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.833+0000 7fde6b7fe700 1 -- 172.21.15.113:0/2473956865 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58011d20 con 0x7fde6c099ba0 2024-01-15T11:26:48.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.834+0000 7fde73100700 1 -- 172.21.15.113:0/2473956865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde6c099ba0 msgr2=0x7fde6c0cfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:48.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.834+0000 7fde73100700 1 --2- 172.21.15.113:0/2473956865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde6c099ba0 0x7fde6c0cfce0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0b6450 tx=0x7fde58009b20 comp rx=0 tx=0).stop 2024-01-15T11:26:48.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.834+0000 7fde73100700 1 -- 172.21.15.113:0/2473956865 shutdown_connections 2024-01-15T11:26:48.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.834+0000 7fde73100700 1 --2- 172.21.15.113:0/2473956865 >> v2:172.21.15.141:3300/0 conn(0x7fde6c161c90 0x7fde6c162070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:48.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.834+0000 7fde73100700 1 --2- 172.21.15.113:0/2473956865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde6c099ba0 0x7fde6c0cfce0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:48.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.834+0000 7fde73100700 1 -- 172.21.15.113:0/2473956865 >> 172.21.15.113:0/2473956865 conn(0x7fde6c0b1e90 msgr2=0x7fde6c0b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:48.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.834+0000 7fde73100700 1 -- 172.21.15.113:0/2473956865 shutdown_connections 2024-01-15T11:26:48.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.834+0000 7fde73100700 1 -- 172.21.15.113:0/2473956865 wait complete. 2024-01-15T11:26:48.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.835+0000 7fde73100700 1 Processor -- start 2024-01-15T11:26:48.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.835+0000 7fde73100700 1 -- start start 2024-01-15T11:26:48.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde73100700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6c099ba0 0x7fde6c15cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:48.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde73100700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde6c15e370 0x7fde6c0cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:48.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde73100700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fde6c161c90 0x7fde6c1e8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:48.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde73100700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0cbfe0 con 0x7fde6c15e370 2024-01-15T11:26:48.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde73100700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0cd820 con 0x7fde6c099ba0 2024-01-15T11:26:48.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde73100700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0b5920 con 0x7fde6c161c90 2024-01-15T11:26:48.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde6bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde6c15e370 0x7fde6c0cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:48.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde6bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde6c15e370 0x7fde6c0cc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36378/0 (socket says 172.21.15.113:36378) 2024-01-15T11:26:48.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde6bfff700 1 -- 172.21.15.113:0/3213633719 learned_addr learned my addr 172.21.15.113:0/3213633719 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:48.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde7169d700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fde6c161c90 0x7fde6c1e8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:48.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde70e9c700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6c099ba0 0x7fde6c15cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:48.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde6bfff700 1 -- 172.21.15.113:0/3213633719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fde6c161c90 msgr2=0x7fde6c1e8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:48.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde6bfff700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fde6c161c90 0x7fde6c1e8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:48.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde6bfff700 1 -- 172.21.15.113:0/3213633719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6c099ba0 msgr2=0x7fde6c15cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:48.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde6bfff700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6c099ba0 0x7fde6c15cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:48.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde6bfff700 1 -- 172.21.15.113:0/3213633719 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde580094a0 con 0x7fde6c15e370 2024-01-15T11:26:48.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde6bfff700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde6c15e370 0x7fde6c0cc400 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fde58000c00 tx=0x7fde58004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:48.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde697fa700 1 -- 172.21.15.113:0/3213633719 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde580038a0 con 0x7fde6c15e370 2024-01-15T11:26:48.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde697fa700 1 -- 172.21.15.113:0/3213633719 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5801f070 con 0x7fde6c15e370 2024-01-15T11:26:48.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde697fa700 1 -- 172.21.15.113:0/3213633719 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58003c40 con 0x7fde6c15e370 2024-01-15T11:26:48.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.836+0000 7fde73100700 1 -- 172.21.15.113:0/3213633719 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6c15d1a0 con 0x7fde6c15e370 2024-01-15T11:26:48.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.837+0000 7fde73100700 1 -- 172.21.15.113:0/3213633719 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6c0cb640 con 0x7fde6c15e370 2024-01-15T11:26:48.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.837+0000 7fde73100700 1 -- 172.21.15.113:0/3213633719 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6c0cc120 con 0x7fde6c15e370 2024-01-15T11:26:48.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.838+0000 7fde697fa700 1 -- 172.21.15.113:0/3213633719 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fde58026480 con 0x7fde6c15e370 2024-01-15T11:26:48.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.838+0000 7fde697fa700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fde5006b500 0x7fde5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:48.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.838+0000 7fde70e9c700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fde5006b500 0x7fde5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:48.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.838+0000 7fde697fa700 1 -- 172.21.15.113:0/3213633719 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde58021070 con 0x7fde6c15e370 2024-01-15T11:26:48.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.839+0000 7fde70e9c700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fde5006b500 0x7fde5006d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fde6000a950 tx=0x7fde60008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:48.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:48.843+0000 7fde697fa700 1 -- 172.21.15.113:0/3213633719 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde6c0cc120 con 0x7fde6c15e370 2024-01-15T11:26:49.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.113+0000 7fde73100700 1 -- 172.21.15.113:0/3213633719 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fde6c0cc120 con 0x7fde6c15e370 2024-01-15T11:26:49.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.114+0000 7fde697fa700 1 -- 172.21.15.113:0/3213633719 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+3294 (secure 0 0 0) 0x7fde6c0cc120 con 0x7fde6c15e370 2024-01-15T11:26:49.113 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:26:49.113 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:26:49.114 INFO:teuthology.orchestra.run.smithi113.stdout: "fsid": "910f8628-cd75-4a37-bf4e-e3603a95290d", 2024-01-15T11:26:49.114 INFO:teuthology.orchestra.run.smithi113.stdout: "health": { 2024-01-15T11:26:49.114 INFO:teuthology.orchestra.run.smithi113.stdout: "status": "HEALTH_OK", 2024-01-15T11:26:49.114 INFO:teuthology.orchestra.run.smithi113.stdout: "checks": {}, 2024-01-15T11:26:49.114 INFO:teuthology.orchestra.run.smithi113.stdout: "mutes": [] 2024-01-15T11:26:49.114 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:49.114 INFO:teuthology.orchestra.run.smithi113.stdout: "election_epoch": 4, 2024-01-15T11:26:49.114 INFO:teuthology.orchestra.run.smithi113.stdout: "quorum": [ 2024-01-15T11:26:49.115 INFO:teuthology.orchestra.run.smithi113.stdout: 0, 2024-01-15T11:26:49.115 INFO:teuthology.orchestra.run.smithi113.stdout: 1, 2024-01-15T11:26:49.115 INFO:teuthology.orchestra.run.smithi113.stdout: 2 2024-01-15T11:26:49.115 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-01-15T11:26:49.115 INFO:teuthology.orchestra.run.smithi113.stdout: "quorum_names": [ 2024-01-15T11:26:49.115 INFO:teuthology.orchestra.run.smithi113.stdout: "a", 2024-01-15T11:26:49.115 INFO:teuthology.orchestra.run.smithi113.stdout: "b", 2024-01-15T11:26:49.115 INFO:teuthology.orchestra.run.smithi113.stdout: "c" 2024-01-15T11:26:49.115 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-01-15T11:26:49.116 INFO:teuthology.orchestra.run.smithi113.stdout: "quorum_age": 118, 2024-01-15T11:26:49.116 INFO:teuthology.orchestra.run.smithi113.stdout: "monmap": { 2024-01-15T11:26:49.116 INFO:teuthology.orchestra.run.smithi113.stdout: "epoch": 1, 2024-01-15T11:26:49.116 INFO:teuthology.orchestra.run.smithi113.stdout: "min_mon_release_name": "reef", 2024-01-15T11:26:49.116 INFO:teuthology.orchestra.run.smithi113.stdout: "num_mons": 3 2024-01-15T11:26:49.116 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:49.116 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap": { 2024-01-15T11:26:49.116 INFO:teuthology.orchestra.run.smithi113.stdout: "epoch": 33, 2024-01-15T11:26:49.116 INFO:teuthology.orchestra.run.smithi113.stdout: "num_osds": 8, 2024-01-15T11:26:49.117 INFO:teuthology.orchestra.run.smithi113.stdout: "num_up_osds": 8, 2024-01-15T11:26:49.117 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_up_since": 1705317901, 2024-01-15T11:26:49.117 INFO:teuthology.orchestra.run.smithi113.stdout: "num_in_osds": 8, 2024-01-15T11:26:49.117 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_in_since": 1705317892, 2024-01-15T11:26:49.117 INFO:teuthology.orchestra.run.smithi113.stdout: "num_remapped_pgs": 0 2024-01-15T11:26:49.117 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:49.117 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap": { 2024-01-15T11:26:49.117 INFO:teuthology.orchestra.run.smithi113.stdout: "pgs_by_state": [ 2024-01-15T11:26:49.117 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:49.117 INFO:teuthology.orchestra.run.smithi113.stdout: "state_name": "active+clean", 2024-01-15T11:26:49.118 INFO:teuthology.orchestra.run.smithi113.stdout: "count": 97 2024-01-15T11:26:49.118 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:49.118 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-01-15T11:26:49.118 INFO:teuthology.orchestra.run.smithi113.stdout: "num_pgs": 97, 2024-01-15T11:26:49.118 INFO:teuthology.orchestra.run.smithi113.stdout: "num_pools": 3, 2024-01-15T11:26:49.118 INFO:teuthology.orchestra.run.smithi113.stdout: "num_objects": 797, 2024-01-15T11:26:49.118 INFO:teuthology.orchestra.run.smithi113.stdout: "data_bytes": 3222059884, 2024-01-15T11:26:49.118 INFO:teuthology.orchestra.run.smithi113.stdout: "bytes_used": 6668513280, 2024-01-15T11:26:49.118 INFO:teuthology.orchestra.run.smithi113.stdout: "bytes_avail": 766425600000, 2024-01-15T11:26:49.118 INFO:teuthology.orchestra.run.smithi113.stdout: "bytes_total": 773094113280, 2024-01-15T11:26:49.119 INFO:teuthology.orchestra.run.smithi113.stdout: "read_bytes_sec": 170, 2024-01-15T11:26:49.119 INFO:teuthology.orchestra.run.smithi113.stdout: "write_bytes_sec": 178576032, 2024-01-15T11:26:49.119 INFO:teuthology.orchestra.run.smithi113.stdout: "read_op_per_sec": 0, 2024-01-15T11:26:49.119 INFO:teuthology.orchestra.run.smithi113.stdout: "write_op_per_sec": 46 2024-01-15T11:26:49.119 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:49.119 INFO:teuthology.orchestra.run.smithi113.stdout: "fsmap": { 2024-01-15T11:26:49.119 INFO:teuthology.orchestra.run.smithi113.stdout: "epoch": 18, 2024-01-15T11:26:49.119 INFO:teuthology.orchestra.run.smithi113.stdout: "id": 2, 2024-01-15T11:26:49.119 INFO:teuthology.orchestra.run.smithi113.stdout: "up": 1, 2024-01-15T11:26:49.119 INFO:teuthology.orchestra.run.smithi113.stdout: "in": 1, 2024-01-15T11:26:49.120 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 1, 2024-01-15T11:26:49.120 INFO:teuthology.orchestra.run.smithi113.stdout: "by_rank": [ 2024-01-15T11:26:49.120 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:26:49.120 INFO:teuthology.orchestra.run.smithi113.stdout: "filesystem_id": 2, 2024-01-15T11:26:49.120 INFO:teuthology.orchestra.run.smithi113.stdout: "rank": 0, 2024-01-15T11:26:49.120 INFO:teuthology.orchestra.run.smithi113.stdout: "name": "a", 2024-01-15T11:26:49.120 INFO:teuthology.orchestra.run.smithi113.stdout: "status": "up:active", 2024-01-15T11:26:49.120 INFO:teuthology.orchestra.run.smithi113.stdout: "gid": 5004 2024-01-15T11:26:49.120 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:49.121 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-01-15T11:26:49.121 INFO:teuthology.orchestra.run.smithi113.stdout: "up:standby": 3 2024-01-15T11:26:49.121 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:49.121 INFO:teuthology.orchestra.run.smithi113.stdout: "mgrmap": { 2024-01-15T11:26:49.121 INFO:teuthology.orchestra.run.smithi113.stdout: "available": true, 2024-01-15T11:26:49.121 INFO:teuthology.orchestra.run.smithi113.stdout: "num_standbys": 1, 2024-01-15T11:26:49.121 INFO:teuthology.orchestra.run.smithi113.stdout: "modules": [ 2024-01-15T11:26:49.121 INFO:teuthology.orchestra.run.smithi113.stdout: "iostat", 2024-01-15T11:26:49.121 INFO:teuthology.orchestra.run.smithi113.stdout: "nfs", 2024-01-15T11:26:49.121 INFO:teuthology.orchestra.run.smithi113.stdout: "restful" 2024-01-15T11:26:49.121 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-01-15T11:26:49.122 INFO:teuthology.orchestra.run.smithi113.stdout: "services": {} 2024-01-15T11:26:49.122 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:49.122 INFO:teuthology.orchestra.run.smithi113.stdout: "servicemap": { 2024-01-15T11:26:49.122 INFO:teuthology.orchestra.run.smithi113.stdout: "epoch": 4, 2024-01-15T11:26:49.122 INFO:teuthology.orchestra.run.smithi113.stdout: "modified": "2024-01-15T11:26:04.118827+0000", 2024-01-15T11:26:49.122 INFO:teuthology.orchestra.run.smithi113.stdout: "services": { 2024-01-15T11:26:49.122 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-01-15T11:26:49.122 INFO:teuthology.orchestra.run.smithi113.stdout: "daemons": { 2024-01-15T11:26:49.122 INFO:teuthology.orchestra.run.smithi113.stdout: "summary": "", 2024-01-15T11:26:49.122 INFO:teuthology.orchestra.run.smithi113.stdout: "a": { 2024-01-15T11:26:49.123 INFO:teuthology.orchestra.run.smithi113.stdout: "start_epoch": 0, 2024-01-15T11:26:49.123 INFO:teuthology.orchestra.run.smithi113.stdout: "start_stamp": "0.000000", 2024-01-15T11:26:49.123 INFO:teuthology.orchestra.run.smithi113.stdout: "gid": 0, 2024-01-15T11:26:49.123 INFO:teuthology.orchestra.run.smithi113.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-15T11:26:49.123 INFO:teuthology.orchestra.run.smithi113.stdout: "metadata": {}, 2024-01-15T11:26:49.123 INFO:teuthology.orchestra.run.smithi113.stdout: "task_status": {} 2024-01-15T11:26:49.123 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:49.123 INFO:teuthology.orchestra.run.smithi113.stdout: "b": { 2024-01-15T11:26:49.123 INFO:teuthology.orchestra.run.smithi113.stdout: "start_epoch": 0, 2024-01-15T11:26:49.123 INFO:teuthology.orchestra.run.smithi113.stdout: "start_stamp": "0.000000", 2024-01-15T11:26:49.124 INFO:teuthology.orchestra.run.smithi113.stdout: "gid": 0, 2024-01-15T11:26:49.124 INFO:teuthology.orchestra.run.smithi113.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-15T11:26:49.124 INFO:teuthology.orchestra.run.smithi113.stdout: "metadata": {}, 2024-01-15T11:26:49.124 INFO:teuthology.orchestra.run.smithi113.stdout: "task_status": {} 2024-01-15T11:26:49.124 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:49.124 INFO:teuthology.orchestra.run.smithi113.stdout: "c": { 2024-01-15T11:26:49.124 INFO:teuthology.orchestra.run.smithi113.stdout: "start_epoch": 0, 2024-01-15T11:26:49.124 INFO:teuthology.orchestra.run.smithi113.stdout: "start_stamp": "0.000000", 2024-01-15T11:26:49.124 INFO:teuthology.orchestra.run.smithi113.stdout: "gid": 0, 2024-01-15T11:26:49.125 INFO:teuthology.orchestra.run.smithi113.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-15T11:26:49.125 INFO:teuthology.orchestra.run.smithi113.stdout: "metadata": {}, 2024-01-15T11:26:49.125 INFO:teuthology.orchestra.run.smithi113.stdout: "task_status": {} 2024-01-15T11:26:49.125 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:49.125 INFO:teuthology.orchestra.run.smithi113.stdout: "d": { 2024-01-15T11:26:49.125 INFO:teuthology.orchestra.run.smithi113.stdout: "start_epoch": 0, 2024-01-15T11:26:49.125 INFO:teuthology.orchestra.run.smithi113.stdout: "start_stamp": "0.000000", 2024-01-15T11:26:49.125 INFO:teuthology.orchestra.run.smithi113.stdout: "gid": 0, 2024-01-15T11:26:49.125 INFO:teuthology.orchestra.run.smithi113.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-15T11:26:49.125 INFO:teuthology.orchestra.run.smithi113.stdout: "metadata": {}, 2024-01-15T11:26:49.125 INFO:teuthology.orchestra.run.smithi113.stdout: "task_status": {} 2024-01-15T11:26:49.126 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:49.126 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:49.126 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:49.126 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:49.126 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:49.126 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_events": {} 2024-01-15T11:26:49.126 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:26:49.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.116+0000 7fde73100700 1 -- 172.21.15.113:0/3213633719 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fde5006b500 msgr2=0x7fde5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:49.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.116+0000 7fde73100700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fde5006b500 0x7fde5006d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fde6000a950 tx=0x7fde60008040 comp rx=0 tx=0).stop 2024-01-15T11:26:49.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.116+0000 7fde73100700 1 -- 172.21.15.113:0/3213633719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde6c15e370 msgr2=0x7fde6c0cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:49.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.116+0000 7fde73100700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde6c15e370 0x7fde6c0cc400 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fde58000c00 tx=0x7fde58004750 comp rx=0 tx=0).stop 2024-01-15T11:26:49.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.116+0000 7fde73100700 1 -- 172.21.15.113:0/3213633719 shutdown_connections 2024-01-15T11:26:49.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.116+0000 7fde73100700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fde5006b500 0x7fde5006d9c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.116+0000 7fde73100700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fde6c161c90 0x7fde6c1e8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.116+0000 7fde73100700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fde6c099ba0 0x7fde6c15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.116+0000 7fde73100700 1 --2- 172.21.15.113:0/3213633719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde6c15e370 0x7fde6c0cc400 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.116+0000 7fde73100700 1 -- 172.21.15.113:0/3213633719 >> 172.21.15.113:0/3213633719 conn(0x7fde6c0b1e90 msgr2=0x7fde6c06e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:49.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.116+0000 7fde73100700 1 -- 172.21.15.113:0/3213633719 shutdown_connections 2024-01-15T11:26:49.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.116+0000 7fde73100700 1 -- 172.21.15.113:0/3213633719 wait complete. 2024-01-15T11:26:49.129 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T11:26:49.130 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T11:26:49.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.254+0000 7f0788982700 1 Processor -- start 2024-01-15T11:26:49.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.254+0000 7f0788982700 1 -- start start 2024-01-15T11:26:49.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.255+0000 7f0788982700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f0784145790 0x7f07840b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:49.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.255+0000 7f0788982700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f078406c100 con 0x7f07840746a0 2024-01-15T11:26:49.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.255+0000 7f0788982700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07840b0660 con 0x7f078406aff0 2024-01-15T11:26:49.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.255+0000 7f0788982700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07840ae0f0 con 0x7f0784145790 2024-01-15T11:26:49.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.255+0000 7f0782d9d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f07840746a0 0x7f07840b3bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:33776/0 (socket says 172.21.15.113:33776) 2024-01-15T11:26:49.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.255+0000 7f0782d9d700 1 -- 172.21.15.113:0/1911268024 learned_addr learned my addr 172.21.15.113:0/1911268024 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:49.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542647276 0 0) 0x7f078406c100 con 0x7f07840746a0 2024-01-15T11:26:49.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07700035b0 con 0x7f07840746a0 2024-01-15T11:26:49.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078259c700 1 --2- 172.21.15.113:0/1911268024 >> v2:172.21.15.141:3300/0 conn(0x7f0784145790 0x7f07840b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:49.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 85218508 0 0) 0x7f07700035b0 con 0x7f07840746a0 2024-01-15T11:26:49.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 >> v2:172.21.15.141:3300/0 conn(0x7f0784145790 msgr2=0x7f07840b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:49.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 --2- 172.21.15.113:0/1911268024 >> v2:172.21.15.141:3300/0 conn(0x7f0784145790 0x7f07840b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 >> v1:172.21.15.141:6789/0 conn(0x7f078406aff0 legacy=0x7f0784073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:49.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078406b4f0 con 0x7f07840746a0 2024-01-15T11:26:49.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f07740019f0 con 0x7f07840746a0 2024-01-15T11:26:49.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078259c700 1 --2- 172.21.15.113:0/1911268024 >> v2:172.21.15.141:3300/0 conn(0x7f0784145790 0x7f07840b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:49.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 >> v1:172.21.15.113:6789/0 conn(0x7f07840746a0 legacy=0x7f07840b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:49.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 --2- 172.21.15.113:0/1911268024 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0770003a00 0x7f0770003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:49.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 --2- 172.21.15.113:0/1911268024 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0770004850 0x7f0770004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:49.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 --2- 172.21.15.113:0/1911268024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0770005210 0x7f077000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:49.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07840ae0f0 con 0x7f0770005210 2024-01-15T11:26:49.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077000c160 con 0x7f0770004850 2024-01-15T11:26:49.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077000c320 con 0x7f0770003a00 2024-01-15T11:26:49.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f0782d9d700 1 --2- 172.21.15.113:0/1911268024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0770005210 0x7f077000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:49.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f078259c700 1 --2- 172.21.15.113:0/1911268024 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0770003a00 0x7f0770003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:49.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f0781d9b700 1 --2- 172.21.15.113:0/1911268024 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0770004850 0x7f0770004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:49.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f0782d9d700 1 -- 172.21.15.113:0/1911268024 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0770003a00 msgr2=0x7f0770003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:49.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f0782d9d700 1 --2- 172.21.15.113:0/1911268024 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0770003a00 0x7f0770003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f0782d9d700 1 -- 172.21.15.113:0/1911268024 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0770004850 msgr2=0x7f0770004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:49.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f0782d9d700 1 --2- 172.21.15.113:0/1911268024 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0770004850 0x7f0770004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f0782d9d700 1 -- 172.21.15.113:0/1911268024 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07700035b0 con 0x7f0770005210 2024-01-15T11:26:49.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f0782d9d700 1 -- 172.21.15.113:0/1911268024 reap_dead start 2024-01-15T11:26:49.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.256+0000 7f0782d9d700 1 --2- 172.21.15.113:0/1911268024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0770005210 0x7f077000ba10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f078409f090 tx=0x7f0770005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0976dea690ea56 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:49.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.257+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077400a2f0 con 0x7f0770005210 2024-01-15T11:26:49.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.257+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0774012600 con 0x7f0770005210 2024-01-15T11:26:49.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.257+0000 7f078159a700 1 -- 172.21.15.113:0/1911268024 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07740129b0 con 0x7f0770005210 2024-01-15T11:26:49.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.257+0000 7f0788982700 1 -- 172.21.15.113:0/1911268024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0770005210 msgr2=0x7f077000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:49.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.257+0000 7f0788982700 1 --2- 172.21.15.113:0/1911268024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0770005210 0x7f077000ba10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f078409f090 tx=0x7f0770005880 comp rx=0 tx=0).stop 2024-01-15T11:26:49.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.257+0000 7f0788982700 1 -- 172.21.15.113:0/1911268024 shutdown_connections 2024-01-15T11:26:49.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.257+0000 7f0788982700 1 --2- 172.21.15.113:0/1911268024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0770005210 0x7f077000ba10 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.257+0000 7f0788982700 1 -- 172.21.15.113:0/1911268024 >> 172.21.15.113:0/1911268024 conn(0x7f07840b3250 msgr2=0x7f07840d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:49.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.257+0000 7f0788982700 1 -- 172.21.15.113:0/1911268024 shutdown_connections 2024-01-15T11:26:49.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.257+0000 7f0788982700 1 -- 172.21.15.113:0/1911268024 wait complete. 2024-01-15T11:26:49.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0788982700 1 Processor -- start 2024-01-15T11:26:49.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0788982700 1 -- start start 2024-01-15T11:26:49.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0788982700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0770005210 0x7f078415e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:49.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0788982700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f078415b250 0x7f0784142500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:49.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0788982700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f078415ca70 0x7f078415a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:49.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0788982700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078415f550 con 0x7f078415b250 2024-01-15T11:26:49.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0788982700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078415ecb0 con 0x7f078415ca70 2024-01-15T11:26:49.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0788982700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078415ef90 con 0x7f0770005210 2024-01-15T11:26:49.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0781d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f078415b250 0x7f0784142500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:49.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0781d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f078415b250 0x7f0784142500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36388/0 (socket says 172.21.15.113:36388) 2024-01-15T11:26:49.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0781d9b700 1 -- 172.21.15.113:0/3359467285 learned_addr learned my addr 172.21.15.113:0/3359467285 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:49.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0782d9d700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f078415ca70 0x7f078415a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:49.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f078259c700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0770005210 0x7f078415e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:49.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0781d9b700 1 -- 172.21.15.113:0/3359467285 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0770005210 msgr2=0x7f078415e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:49.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0781d9b700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0770005210 0x7f078415e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0781d9b700 1 -- 172.21.15.113:0/3359467285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f078415ca70 msgr2=0x7f078415a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:49.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0781d9b700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f078415ca70 0x7f078415a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0781d9b700 1 -- 172.21.15.113:0/3359467285 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078415f270 con 0x7f078415b250 2024-01-15T11:26:49.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0782d9d700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f078415ca70 0x7f078415a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:49.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0781d9b700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f078415b250 0x7f0784142500 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f076c00f820 tx=0x7f076c00fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:49.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f077affd700 1 -- 172.21.15.113:0/3359467285 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c0052e0 con 0x7f078415b250 2024-01-15T11:26:49.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f077affd700 1 -- 172.21.15.113:0/3359467285 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c005460 con 0x7f078415b250 2024-01-15T11:26:49.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0788982700 1 -- 172.21.15.113:0/3359467285 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f078415c550 con 0x7f078415b250 2024-01-15T11:26:49.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f077affd700 1 -- 172.21.15.113:0/3359467285 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c0038c0 con 0x7f078415b250 2024-01-15T11:26:49.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.260+0000 7f0788982700 1 -- 172.21.15.113:0/3359467285 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0784142c20 con 0x7f078415b250 2024-01-15T11:26:49.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.261+0000 7f0788982700 1 -- 172.21.15.113:0/3359467285 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078415c3b0 con 0x7f078415b250 2024-01-15T11:26:49.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.261+0000 7f077affd700 1 -- 172.21.15.113:0/3359467285 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f076c003a20 con 0x7f078415b250 2024-01-15T11:26:49.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.262+0000 7f077affd700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f076406b530 0x7f076406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:49.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.262+0000 7f077affd700 1 -- 172.21.15.113:0/3359467285 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f076c089ae0 con 0x7f078415b250 2024-01-15T11:26:49.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.262+0000 7f078259c700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f076406b530 0x7f076406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:49.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.262+0000 7f078259c700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f076406b530 0x7f076406d9f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f077c004470 tx=0x7f077c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:49.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.266+0000 7f077affd700 1 -- 172.21.15.113:0/3359467285 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f078415c3b0 con 0x7f078415b250 2024-01-15T11:26:49.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.498+0000 7f0788982700 1 -- 172.21.15.113:0/3359467285 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f078406bab0 con 0x7f076406b530 2024-01-15T11:26:49.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.505+0000 7f077affd700 1 -- 172.21.15.113:0/3359467285 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f077c004550 con 0x7f076406b530 2024-01-15T11:26:49.504 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:26:49.504 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-15T11:26:49.505 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-15T11:26:49.505 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-15T11:26:49.505 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-15T11:26:49.505 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-15T11:26:49.505 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-15T11:26:49.505 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:49.505 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-15T11:26:49.505 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 0.0, 2024-01-15T11:26:49.505 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "0 B/3 GB", 2024-01-15T11:26:49.505 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "0/3", 2024-01-15T11:26:49.505 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-15T11:26:49.505 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-15T11:26:49.506 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:49.506 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:49.506 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:26:49.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.508+0000 7f0788982700 1 -- 172.21.15.113:0/3359467285 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f076406b530 msgr2=0x7f076406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:49.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.508+0000 7f0788982700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f076406b530 0x7f076406d9f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f077c004470 tx=0x7f077c008040 comp rx=0 tx=0).stop 2024-01-15T11:26:49.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.508+0000 7f0788982700 1 -- 172.21.15.113:0/3359467285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f078415b250 msgr2=0x7f0784142500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:49.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.508+0000 7f0788982700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f078415b250 0x7f0784142500 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f076c00f820 tx=0x7f076c00fbe0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.508+0000 7f0788982700 1 -- 172.21.15.113:0/3359467285 shutdown_connections 2024-01-15T11:26:49.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.508+0000 7f0788982700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f076406b530 0x7f076406d9f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.508+0000 7f0788982700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f078415ca70 0x7f078415a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.508+0000 7f0788982700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f0770005210 0x7f078415e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.508+0000 7f0788982700 1 --2- 172.21.15.113:0/3359467285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f078415b250 0x7f0784142500 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:49.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.508+0000 7f0788982700 1 -- 172.21.15.113:0/3359467285 >> 172.21.15.113:0/3359467285 conn(0x7f07840b3250 msgr2=0x7f07840b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:49.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.508+0000 7f0788982700 1 -- 172.21.15.113:0/3359467285 shutdown_connections 2024-01-15T11:26:49.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:49.508+0000 7f0788982700 1 -- 172.21.15.113:0/3359467285 wait complete. 2024-01-15T11:26:49.519 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T11:26:50.519 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T11:26:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.661+0000 7fee6b7c6700 1 Processor -- start 2024-01-15T11:26:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.661+0000 7fee6b7c6700 1 -- start start 2024-01-15T11:26:50.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee6b7c6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fee6415e110 0x7fee6406f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:50.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee6b7c6700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fee640afb30 0x7fee6406aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:50.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee6b7c6700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee640affe0 con 0x7fee6406b530 2024-01-15T11:26:50.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee6b7c6700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6406d980 con 0x7fee6415e110 2024-01-15T11:26:50.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee6b7c6700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee640b0260 con 0x7fee640afb30 2024-01-15T11:26:50.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee69562700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fee6415e110 0x7fee6406f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:50.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee69d63700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fee6406b530 0x7fee6406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:33778/0 (socket says 172.21.15.113:33778) 2024-01-15T11:26:50.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee69d63700 1 -- 172.21.15.113:0/353680921 learned_addr learned my addr 172.21.15.113:0/353680921 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:50.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee68d61700 1 --2- 172.21.15.113:0/353680921 >> v2:172.21.15.141:3300/0 conn(0x7fee640afb30 0x7fee6406aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:50.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee69562700 1 -- 172.21.15.113:0/353680921 >> v2:172.21.15.141:3300/0 conn(0x7fee640afb30 msgr2=0x7fee6406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:50.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee69562700 1 --2- 172.21.15.113:0/353680921 >> v2:172.21.15.141:3300/0 conn(0x7fee640afb30 0x7fee6406aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:50.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee5bfff700 1 -- 172.21.15.113:0/353680921 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913145997 0 0) 0x7fee640affe0 con 0x7fee6406b530 2024-01-15T11:26:50.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee69562700 1 -- 172.21.15.113:0/353680921 >> v1:172.21.15.113:6789/0 conn(0x7fee6406b530 legacy=0x7fee6406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:50.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee69562700 1 -- 172.21.15.113:0/353680921 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee640b0540 con 0x7fee6415e110 2024-01-15T11:26:50.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee68d61700 1 --2- 172.21.15.113:0/353680921 >> v2:172.21.15.141:3300/0 conn(0x7fee640afb30 0x7fee6406aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:50.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee69562700 1 --2- 172.21.15.113:0/353680921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6415e110 0x7fee6406f680 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fee640b3990 tx=0x7fee54009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4cff14b2c98edee server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:50.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee5bfff700 1 -- 172.21.15.113:0/353680921 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee54011830 con 0x7fee6415e110 2024-01-15T11:26:50.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee5bfff700 1 -- 172.21.15.113:0/353680921 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee54011990 con 0x7fee6415e110 2024-01-15T11:26:50.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.662+0000 7fee5bfff700 1 -- 172.21.15.113:0/353680921 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee54011d20 con 0x7fee6415e110 2024-01-15T11:26:50.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.663+0000 7fee6b7c6700 1 -- 172.21.15.113:0/353680921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6415e110 msgr2=0x7fee6406f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:50.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.663+0000 7fee6b7c6700 1 --2- 172.21.15.113:0/353680921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6415e110 0x7fee6406f680 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fee640b3990 tx=0x7fee54009b20 comp rx=0 tx=0).stop 2024-01-15T11:26:50.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.663+0000 7fee6b7c6700 1 -- 172.21.15.113:0/353680921 shutdown_connections 2024-01-15T11:26:50.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.663+0000 7fee6b7c6700 1 --2- 172.21.15.113:0/353680921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6415e110 0x7fee6406f680 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:50.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.663+0000 7fee6b7c6700 1 --2- 172.21.15.113:0/353680921 >> v2:172.21.15.141:3300/0 conn(0x7fee640afb30 0x7fee6406aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:50.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.663+0000 7fee6b7c6700 1 -- 172.21.15.113:0/353680921 >> 172.21.15.113:0/353680921 conn(0x7fee640b3250 msgr2=0x7fee640d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:50.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.663+0000 7fee6b7c6700 1 -- 172.21.15.113:0/353680921 shutdown_connections 2024-01-15T11:26:50.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.663+0000 7fee6b7c6700 1 -- 172.21.15.113:0/353680921 wait complete. 2024-01-15T11:26:50.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.664+0000 7fee6b7c6700 1 Processor -- start 2024-01-15T11:26:50.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.664+0000 7fee6b7c6700 1 -- start start 2024-01-15T11:26:50.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee6b7c6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6406b530 0x7fee640cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:50.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee6b7c6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee640afb30 0x7fee6414f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:50.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee6b7c6700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fee6415e110 0x7fee640cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:50.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee6b7c6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee640cde10 con 0x7fee6406b530 2024-01-15T11:26:50.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee6b7c6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6414fc30 con 0x7fee640afb30 2024-01-15T11:26:50.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee6b7c6700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee640c9420 con 0x7fee6415e110 2024-01-15T11:26:50.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee69562700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6406b530 0x7fee640cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:50.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee69562700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6406b530 0x7fee640cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36394/0 (socket says 172.21.15.113:36394) 2024-01-15T11:26:50.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee69562700 1 -- 172.21.15.113:0/3244495114 learned_addr learned my addr 172.21.15.113:0/3244495114 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:50.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee68d61700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee640afb30 0x7fee6414f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:50.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee69d63700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fee6415e110 0x7fee640cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:50.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee69562700 1 -- 172.21.15.113:0/3244495114 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fee6415e110 msgr2=0x7fee640cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:50.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee69562700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fee6415e110 0x7fee640cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:50.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee69562700 1 -- 172.21.15.113:0/3244495114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee640afb30 msgr2=0x7fee6414f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:50.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee69562700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee640afb30 0x7fee6414f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:50.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.665+0000 7fee69562700 1 -- 172.21.15.113:0/3244495114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee540094a0 con 0x7fee6406b530 2024-01-15T11:26:50.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.666+0000 7fee69562700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6406b530 0x7fee640cd8d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fee54000c00 tx=0x7fee54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:50.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.666+0000 7fee59ffb700 1 -- 172.21.15.113:0/3244495114 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee540038a0 con 0x7fee6406b530 2024-01-15T11:26:50.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.666+0000 7fee59ffb700 1 -- 172.21.15.113:0/3244495114 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5401e070 con 0x7fee6406b530 2024-01-15T11:26:50.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.666+0000 7fee6b7c6700 1 -- 172.21.15.113:0/3244495114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee640c9700 con 0x7fee6406b530 2024-01-15T11:26:50.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.666+0000 7fee59ffb700 1 -- 172.21.15.113:0/3244495114 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee54003c40 con 0x7fee6406b530 2024-01-15T11:26:50.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.666+0000 7fee6b7c6700 1 -- 172.21.15.113:0/3244495114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee640c9cc0 con 0x7fee6406b530 2024-01-15T11:26:50.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.667+0000 7fee6b7c6700 1 -- 172.21.15.113:0/3244495114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee64069020 con 0x7fee6406b530 2024-01-15T11:26:50.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.667+0000 7fee59ffb700 1 -- 172.21.15.113:0/3244495114 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fee5401a420 con 0x7fee6406b530 2024-01-15T11:26:50.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.668+0000 7fee59ffb700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fee4806b500 0x7fee4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:50.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.668+0000 7fee59ffb700 1 -- 172.21.15.113:0/3244495114 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee5408afa0 con 0x7fee6406b530 2024-01-15T11:26:50.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.668+0000 7fee68d61700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fee4806b500 0x7fee4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:50.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.668+0000 7fee68d61700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fee4806b500 0x7fee4806d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fee640d04c0 tx=0x7fee60008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:50.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.672+0000 7fee59ffb700 1 -- 172.21.15.113:0/3244495114 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee540bde90 con 0x7fee6406b530 2024-01-15T11:26:50.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.985+0000 7fee6b7c6700 1 -- 172.21.15.113:0/3244495114 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fee6406df00 con 0x7fee4806b500 2024-01-15T11:26:50.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.992+0000 7fee59ffb700 1 -- 172.21.15.113:0/3244495114 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7fee6000c050 con 0x7fee4806b500 2024-01-15T11:26:50.991 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:26:50.991 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-15T11:26:50.991 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-15T11:26:50.991 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-15T11:26:50.991 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-15T11:26:50.991 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-15T11:26:50.991 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-15T11:26:50.992 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:50.992 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-15T11:26:50.992 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 0.0, 2024-01-15T11:26:50.992 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "0 B/3 GB", 2024-01-15T11:26:50.992 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "0/3", 2024-01-15T11:26:50.992 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-15T11:26:50.992 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-15T11:26:50.992 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:50.992 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:50.992 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:26:50.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.995+0000 7fee6b7c6700 1 -- 172.21.15.113:0/3244495114 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fee4806b500 msgr2=0x7fee4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:50.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.995+0000 7fee6b7c6700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fee4806b500 0x7fee4806d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fee640d04c0 tx=0x7fee60008040 comp rx=0 tx=0).stop 2024-01-15T11:26:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.995+0000 7fee6b7c6700 1 -- 172.21.15.113:0/3244495114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6406b530 msgr2=0x7fee640cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.995+0000 7fee6b7c6700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6406b530 0x7fee640cd8d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fee54000c00 tx=0x7fee54004750 comp rx=0 tx=0).stop 2024-01-15T11:26:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.995+0000 7fee6b7c6700 1 -- 172.21.15.113:0/3244495114 shutdown_connections 2024-01-15T11:26:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.995+0000 7fee6b7c6700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fee4806b500 0x7fee4806d9c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.995+0000 7fee6b7c6700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fee6415e110 0x7fee640cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.995+0000 7fee6b7c6700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6406b530 0x7fee640cd8d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.995+0000 7fee6b7c6700 1 --2- 172.21.15.113:0/3244495114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee640afb30 0x7fee6414f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.995+0000 7fee6b7c6700 1 -- 172.21.15.113:0/3244495114 >> 172.21.15.113:0/3244495114 conn(0x7fee640b3250 msgr2=0x7fee640add40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.995+0000 7fee6b7c6700 1 -- 172.21.15.113:0/3244495114 shutdown_connections 2024-01-15T11:26:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:50.995+0000 7fee6b7c6700 1 -- 172.21.15.113:0/3244495114 wait complete. 2024-01-15T11:26:51.028 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T11:26:52.029 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T11:26:52.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.262+0000 7f2087e23700 1 Processor -- start 2024-01-15T11:26:52.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.262+0000 7f2087e23700 1 -- start start 2024-01-15T11:26:52.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.264+0000 7f2087e23700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f20800b26f0 0x7f208006edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:52.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.264+0000 7f2087e23700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208015cca0 con 0x7f208006b800 2024-01-15T11:26:52.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.264+0000 7f2087e23700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208015c3b0 con 0x7f20801599b0 2024-01-15T11:26:52.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.264+0000 7f2087e23700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208015c650 con 0x7f20800b26f0 2024-01-15T11:26:52.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.264+0000 7f20863c0700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f20800b26f0 0x7f208006edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:52.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.264+0000 7f20853be700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f208006b800 0x7f208015f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:33786/0 (socket says 172.21.15.113:33786) 2024-01-15T11:26:52.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.264+0000 7f2085bbf700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7f20801599b0 0x7f20800b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:36998/0 (socket says 172.21.15.113:36998) 2024-01-15T11:26:52.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.264+0000 7f20863c0700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7f20800b26f0 0x7f208006edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:36626/0 (socket says 172.21.15.113:36626) 2024-01-15T11:26:52.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.264+0000 7f20853be700 1 -- 172.21.15.113:0/1722561411 learned_addr learned my addr 172.21.15.113:0/1722561411 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:52.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.265+0000 7f2084bbd700 1 -- 172.21.15.113:0/1722561411 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969555692 0 0) 0x7f208015cca0 con 0x7f208006b800 2024-01-15T11:26:52.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.265+0000 7f2084bbd700 1 -- 172.21.15.113:0/1722561411 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f206c0035b0 con 0x7f208006b800 2024-01-15T11:26:52.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.265+0000 7f20863c0700 1 -- 172.21.15.113:0/1722561411 >> v1:172.21.15.141:6789/0 conn(0x7f20801599b0 legacy=0x7f20800b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:52.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.265+0000 7f2084bbd700 1 -- 172.21.15.113:0/1722561411 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710795385 0 0) 0x7f208015c3b0 con 0x7f20801599b0 2024-01-15T11:26:52.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.265+0000 7f20863c0700 1 -- 172.21.15.113:0/1722561411 >> v1:172.21.15.113:6789/0 conn(0x7f208006b800 legacy=0x7f208015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:52.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.265+0000 7f20863c0700 1 -- 172.21.15.113:0/1722561411 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208015c930 con 0x7f20800b26f0 2024-01-15T11:26:52.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.265+0000 7f20863c0700 1 --2- 172.21.15.113:0/1722561411 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20800b26f0 0x7f208006edd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f208015c2e0 tx=0x7f207800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89d09b5995ce0d47 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:52.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.265+0000 7f2084bbd700 1 -- 172.21.15.113:0/1722561411 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2078016070 con 0x7f20800b26f0 2024-01-15T11:26:52.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.266+0000 7f2084bbd700 1 -- 172.21.15.113:0/1722561411 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207800bb20 con 0x7f20800b26f0 2024-01-15T11:26:52.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.266+0000 7f2084bbd700 1 -- 172.21.15.113:0/1722561411 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207800e070 con 0x7f20800b26f0 2024-01-15T11:26:52.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.267+0000 7f2087e23700 1 -- 172.21.15.113:0/1722561411 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20800b26f0 msgr2=0x7f208006edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:52.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.267+0000 7f2087e23700 1 --2- 172.21.15.113:0/1722561411 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20800b26f0 0x7f208006edd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f208015c2e0 tx=0x7f207800b330 comp rx=0 tx=0).stop 2024-01-15T11:26:52.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.267+0000 7f2087e23700 1 -- 172.21.15.113:0/1722561411 shutdown_connections 2024-01-15T11:26:52.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.267+0000 7f2087e23700 1 --2- 172.21.15.113:0/1722561411 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20800b26f0 0x7f208006edd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:52.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.267+0000 7f2087e23700 1 -- 172.21.15.113:0/1722561411 >> 172.21.15.113:0/1722561411 conn(0x7f2080074170 msgr2=0x7f208006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.268+0000 7f2087e23700 1 -- 172.21.15.113:0/1722561411 shutdown_connections 2024-01-15T11:26:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.268+0000 7f2087e23700 1 -- 172.21.15.113:0/1722561411 wait complete. 2024-01-15T11:26:52.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.271+0000 7f2087e23700 1 Processor -- start 2024-01-15T11:26:52.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.271+0000 7f2087e23700 1 -- start start 2024-01-15T11:26:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f2087e23700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f208006b800 0x7f20801f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f2087e23700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f20800b26f0 0x7f20800ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f2087e23700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20801599b0 0x7f20801f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f2087e23700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20801f4110 con 0x7f20801599b0 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f2087e23700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20801f43f0 con 0x7f208006b800 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f2087e23700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20801f46d0 con 0x7f20800b26f0 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f20863c0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20801599b0 0x7f20801f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f20863c0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20801599b0 0x7f20801f4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36408/0 (socket says 172.21.15.113:36408) 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f20863c0700 1 -- 172.21.15.113:0/1487282945 learned_addr learned my addr 172.21.15.113:0/1487282945 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f20853be700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f20800b26f0 0x7f20800ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f2085bbf700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f208006b800 0x7f20801f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f20863c0700 1 -- 172.21.15.113:0/1487282945 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f20800b26f0 msgr2=0x7f20800ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f20863c0700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f20800b26f0 0x7f20800ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f20863c0700 1 -- 172.21.15.113:0/1487282945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f208006b800 msgr2=0x7f20801f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f20863c0700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f208006b800 0x7f20801f3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f20863c0700 1 -- 172.21.15.113:0/1487282945 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2078009a70 con 0x7f20801599b0 2024-01-15T11:26:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.272+0000 7f2085bbf700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f208006b800 0x7f20801f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.273+0000 7f20863c0700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20801599b0 0x7f20801f4ca0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f207800b370 tx=0x7f2078004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.273+0000 7f20767fc700 1 -- 172.21.15.113:0/1487282945 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2078016040 con 0x7f20801599b0 2024-01-15T11:26:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.273+0000 7f20767fc700 1 -- 172.21.15.113:0/1487282945 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207801b440 con 0x7f20801599b0 2024-01-15T11:26:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.273+0000 7f2087e23700 1 -- 172.21.15.113:0/1487282945 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20800b39c0 con 0x7f20801599b0 2024-01-15T11:26:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.273+0000 7f20767fc700 1 -- 172.21.15.113:0/1487282945 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207801b7f0 con 0x7f20801599b0 2024-01-15T11:26:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.273+0000 7f2087e23700 1 -- 172.21.15.113:0/1487282945 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20801572f0 con 0x7f20801599b0 2024-01-15T11:26:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.275+0000 7f2087e23700 1 -- 172.21.15.113:0/1487282945 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20680052f0 con 0x7f20801599b0 2024-01-15T11:26:52.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.277+0000 7f20767fc700 1 -- 172.21.15.113:0/1487282945 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f207801b950 con 0x7f20801599b0 2024-01-15T11:26:52.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.277+0000 7f20767fc700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f206406b5d0 0x7f206406da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:52.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.277+0000 7f20767fc700 1 -- 172.21.15.113:0/1487282945 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f207808b8e0 con 0x7f20801599b0 2024-01-15T11:26:52.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.277+0000 7f2085bbf700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f206406b5d0 0x7f206406da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:52.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.278+0000 7f2085bbf700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f206406b5d0 0x7f206406da90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f207c000f60 tx=0x7f207c007cc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:52.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.281+0000 7f20767fc700 1 -- 172.21.15.113:0/1487282945 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2078056550 con 0x7f20801599b0 2024-01-15T11:26:52.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.548+0000 7f2087e23700 1 -- 172.21.15.113:0/1487282945 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2068000bc0 con 0x7f206406b5d0 2024-01-15T11:26:52.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.554+0000 7f20767fc700 1 -- 172.21.15.113:0/1487282945 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f2068000bc0 con 0x7f206406b5d0 2024-01-15T11:26:52.552 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:26:52.552 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-15T11:26:52.552 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 33.33333333333333, 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "1/3", 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:52.553 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:26:52.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.556+0000 7f2087e23700 1 -- 172.21.15.113:0/1487282945 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f206406b5d0 msgr2=0x7f206406da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:52.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.556+0000 7f2087e23700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f206406b5d0 0x7f206406da90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f207c000f60 tx=0x7f207c007cc0 comp rx=0 tx=0).stop 2024-01-15T11:26:52.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.556+0000 7f2087e23700 1 -- 172.21.15.113:0/1487282945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20801599b0 msgr2=0x7f20801f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:52.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.556+0000 7f2087e23700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20801599b0 0x7f20801f4ca0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f207800b370 tx=0x7f2078004ad0 comp rx=0 tx=0).stop 2024-01-15T11:26:52.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.556+0000 7f2087e23700 1 -- 172.21.15.113:0/1487282945 shutdown_connections 2024-01-15T11:26:52.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.556+0000 7f2087e23700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f206406b5d0 0x7f206406da90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:52.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.556+0000 7f2087e23700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20801599b0 0x7f20801f4ca0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:52.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.556+0000 7f2087e23700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f208006b800 0x7f20801f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:52.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.556+0000 7f2087e23700 1 --2- 172.21.15.113:0/1487282945 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f20800b26f0 0x7f20800ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:52.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.556+0000 7f2087e23700 1 -- 172.21.15.113:0/1487282945 >> 172.21.15.113:0/1487282945 conn(0x7f2080074170 msgr2=0x7f208006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:52.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.556+0000 7f2087e23700 1 -- 172.21.15.113:0/1487282945 shutdown_connections 2024-01-15T11:26:52.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:52.556+0000 7f2087e23700 1 -- 172.21.15.113:0/1487282945 wait complete. 2024-01-15T11:26:52.566 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T11:26:53.568 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T11:26:53.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.708+0000 7ffaf5c82700 1 Processor -- start 2024-01-15T11:26:53.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.708+0000 7ffaf5c82700 1 -- start start 2024-01-15T11:26:53.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.708+0000 7ffaf5c82700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ffaf015ae00 0x7ffaf006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.708+0000 7ffaf5c82700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffaf00d05b0 0x7ffaf015f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.708+0000 7ffaf5c82700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf00b3d60 con 0x7ffaf00add40 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.708+0000 7ffaf5c82700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf015f8b0 con 0x7ffaf00d05b0 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.708+0000 7ffaf5c82700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf015b1e0 con 0x7ffaf015ae00 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaeffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffaf00d05b0 0x7ffaf015f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaeeffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ffaf00add40 0x7ffaf00cfea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:33802/0 (socket says 172.21.15.113:33802) 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaeeffd700 1 -- 172.21.15.113:0/3647188604 learned_addr learned my addr 172.21.15.113:0/3647188604 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaeffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffaf00d05b0 0x7ffaf015f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36420/0 (socket says 172.21.15.113:36420) 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaef7fe700 1 --2- 172.21.15.113:0/3647188604 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ffaf015ae00 0x7ffaf006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaeffff700 1 -- 172.21.15.113:0/3647188604 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ffaf015ae00 msgr2=0x7ffaf006dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaeffff700 1 --2- 172.21.15.113:0/3647188604 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ffaf015ae00 0x7ffaf006dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaeffff700 1 -- 172.21.15.113:0/3647188604 >> v1:172.21.15.113:6789/0 conn(0x7ffaf00add40 legacy=0x7ffaf00cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:53.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaee7fc700 1 -- 172.21.15.113:0/3647188604 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647456464 0 0) 0x7ffaf00b3d60 con 0x7ffaf00add40 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaeffff700 1 -- 172.21.15.113:0/3647188604 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf00b4040 con 0x7ffaf00d05b0 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaeffff700 1 --2- 172.21.15.113:0/3647188604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaf00d05b0 0x7ffaf015f280 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009dc0 tx=0x7ffae800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7671c99ef71266b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaee7fc700 1 -- 172.21.15.113:0/3647188604 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae800e070 con 0x7ffaf00d05b0 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaee7fc700 1 -- 172.21.15.113:0/3647188604 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae800bb90 con 0x7ffaf00d05b0 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.709+0000 7ffaee7fc700 1 -- 172.21.15.113:0/3647188604 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8012430 con 0x7ffaf00d05b0 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.710+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3647188604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaf00d05b0 msgr2=0x7ffaf015f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.710+0000 7ffaf5c82700 1 --2- 172.21.15.113:0/3647188604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaf00d05b0 0x7ffaf015f280 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009dc0 tx=0x7ffae800b330 comp rx=0 tx=0).stop 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.710+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3647188604 shutdown_connections 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.710+0000 7ffaf5c82700 1 --2- 172.21.15.113:0/3647188604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaf00d05b0 0x7ffaf015f280 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.710+0000 7ffaf5c82700 1 --2- 172.21.15.113:0/3647188604 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ffaf015ae00 0x7ffaf006dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.710+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3647188604 >> 172.21.15.113:0/3647188604 conn(0x7ffaf0072b20 msgr2=0x7ffaf006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.710+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3647188604 shutdown_connections 2024-01-15T11:26:53.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.710+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3647188604 wait complete. 2024-01-15T11:26:53.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.711+0000 7ffaf5c82700 1 Processor -- start 2024-01-15T11:26:53.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.711+0000 7ffaf5c82700 1 -- start start 2024-01-15T11:26:53.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaf5c82700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaf00add40 0x7ffaf014bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:53.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaf5c82700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00d05b0 0x7ffaf00af710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:53.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaf5c82700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ffaf015ae00 0x7ffaf00afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:53.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaf5c82700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf009d210 con 0x7ffaf00add40 2024-01-15T11:26:53.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaf5c82700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf009c600 con 0x7ffaf00d05b0 2024-01-15T11:26:53.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaf5c82700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf009c8e0 con 0x7ffaf015ae00 2024-01-15T11:26:53.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaef7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaf00add40 0x7ffaf014bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:53.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaef7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaf00add40 0x7ffaf014bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36434/0 (socket says 172.21.15.113:36434) 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaef7fe700 1 -- 172.21.15.113:0/3597937549 learned_addr learned my addr 172.21.15.113:0/3597937549 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaeffff700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ffaf015ae00 0x7ffaf00afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaeeffd700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00d05b0 0x7ffaf00af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaef7fe700 1 -- 172.21.15.113:0/3597937549 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ffaf015ae00 msgr2=0x7ffaf00afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaef7fe700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ffaf015ae00 0x7ffaf00afc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaef7fe700 1 -- 172.21.15.113:0/3597937549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00d05b0 msgr2=0x7ffaf00af710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaef7fe700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00d05b0 0x7ffaf00af710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaef7fe700 1 -- 172.21.15.113:0/3597937549 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae8009a70 con 0x7ffaf00add40 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.712+0000 7ffaef7fe700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaf00add40 0x7ffaf014bca0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ffad800d800 tx=0x7ffad800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.713+0000 7ffae7fff700 1 -- 172.21.15.113:0/3597937549 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad80125f0 con 0x7ffaf00add40 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.713+0000 7ffae7fff700 1 -- 172.21.15.113:0/3597937549 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad8012750 con 0x7ffaf00add40 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.713+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3597937549 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf009cbc0 con 0x7ffaf00add40 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.713+0000 7ffae7fff700 1 -- 172.21.15.113:0/3597937549 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8012b00 con 0x7ffaf00add40 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.713+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3597937549 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf014c1e0 con 0x7ffaf00add40 2024-01-15T11:26:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.714+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3597937549 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf009d4f0 con 0x7ffaf00add40 2024-01-15T11:26:53.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.714+0000 7ffae7fff700 1 -- 172.21.15.113:0/3597937549 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7ffad8012c60 con 0x7ffaf00add40 2024-01-15T11:26:53.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.714+0000 7ffae7fff700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ffad006b500 0x7ffad006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:53.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.715+0000 7ffae7fff700 1 -- 172.21.15.113:0/3597937549 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffad808a250 con 0x7ffaf00add40 2024-01-15T11:26:53.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.715+0000 7ffaeeffd700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ffad006b500 0x7ffad006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:53.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.715+0000 7ffaeeffd700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ffad006b500 0x7ffad006d9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ffaf006f8c0 tx=0x7ffae0007be0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:53.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.719+0000 7ffae7fff700 1 -- 172.21.15.113:0/3597937549 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffad8054e40 con 0x7ffaf00add40 2024-01-15T11:26:53.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.949+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3597937549 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7ffaf009ee00 con 0x7ffad006b500 2024-01-15T11:26:53.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.954+0000 7ffae7fff700 1 -- 172.21.15.113:0/3597937549 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7ffae0005aa0 con 0x7ffad006b500 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 33.33333333333333, 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "1/3", 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-15T11:26:53.954 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-15T11:26:53.955 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:53.955 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:53.955 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:26:53.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.956+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3597937549 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ffad006b500 msgr2=0x7ffad006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:53.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.956+0000 7ffaf5c82700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ffad006b500 0x7ffad006d9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ffaf006f8c0 tx=0x7ffae0007be0 comp rx=0 tx=0).stop 2024-01-15T11:26:53.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.956+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3597937549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaf00add40 msgr2=0x7ffaf014bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:53.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.956+0000 7ffaf5c82700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaf00add40 0x7ffaf014bca0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ffad800d800 tx=0x7ffad800dbc0 comp rx=0 tx=0).stop 2024-01-15T11:26:53.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.956+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3597937549 shutdown_connections 2024-01-15T11:26:53.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.956+0000 7ffaf5c82700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7ffad006b500 0x7ffad006d9c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:53.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.956+0000 7ffaf5c82700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7ffaf015ae00 0x7ffaf00afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:53.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.956+0000 7ffaf5c82700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaf00add40 0x7ffaf014bca0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:53.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.956+0000 7ffaf5c82700 1 --2- 172.21.15.113:0/3597937549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00d05b0 0x7ffaf00af710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:53.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.956+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3597937549 >> 172.21.15.113:0/3597937549 conn(0x7ffaf0072b20 msgr2=0x7ffaf006df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:53.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.957+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3597937549 shutdown_connections 2024-01-15T11:26:53.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:53.957+0000 7ffaf5c82700 1 -- 172.21.15.113:0/3597937549 wait complete. 2024-01-15T11:26:53.968 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T11:26:54.969 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T11:26:55.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.179+0000 7f3b20c19700 1 Processor -- start 2024-01-15T11:26:55.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.179+0000 7f3b20c19700 1 -- start start 2024-01-15T11:26:55.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.181+0000 7f3b20c19700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0d0850 0x7f3b1c0d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:55.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.181+0000 7f3b20c19700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b1c0d1170 con 0x7f3b1c0b9be0 2024-01-15T11:26:55.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.181+0000 7f3b20c19700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b1c0b8580 con 0x7f3b1c0b9760 2024-01-15T11:26:55.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.181+0000 7f3b20c19700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0b8860 con 0x7f3b1c0d0850 2024-01-15T11:26:55.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.181+0000 7f3b1ad9d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f3b1c0b9be0 0x7f3b1c0d29b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51916/0 (socket says 172.21.15.113:51916) 2024-01-15T11:26:55.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.181+0000 7f3b1ad9d700 1 -- 172.21.15.113:0/501533302 learned_addr learned my addr 172.21.15.113:0/501533302 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:55.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.181+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570231092 0 0) 0x7f3b1c0d1170 con 0x7f3b1c0b9be0 2024-01-15T11:26:55.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.181+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b040035b0 con 0x7f3b1c0b9be0 2024-01-15T11:26:55.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.181+0000 7f3b1a59c700 1 --2- 172.21.15.113:0/501533302 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0d0850 0x7f3b1c0d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:55.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4083347603 0 0) 0x7f3b040035b0 con 0x7f3b1c0b9be0 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0d0850 msgr2=0x7f3b1c0d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 --2- 172.21.15.113:0/501533302 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0d0850 0x7f3b1c0d0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 >> v1:172.21.15.141:6789/0 conn(0x7f3b1c0b9760 legacy=0x7f3b1c0d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c0b8b40 con 0x7f3b1c0b9be0 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7f3b14001670 con 0x7f3b1c0b9be0 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 >> v1:172.21.15.113:6789/0 conn(0x7f3b1c0b9be0 legacy=0x7f3b1c0d29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 --2- 172.21.15.113:0/501533302 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0d0850 0x7f3b04003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 --2- 172.21.15.113:0/501533302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b04004090 0x7f3b04004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 --2- 172.21.15.113:0/501533302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b04004d90 0x7f3b04005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0b8860 con 0x7f3b04004d90 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0400bc00 con 0x7f3b04004090 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0400bdc0 con 0x7f3b1c0d0850 2024-01-15T11:26:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1ad9d700 1 --2- 172.21.15.113:0/501533302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b04004d90 0x7f3b04005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1ad9d700 1 -- 172.21.15.113:0/501533302 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0d0850 msgr2=0x7f3b04003a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1ad9d700 1 --2- 172.21.15.113:0/501533302 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0d0850 0x7f3b04003a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1ad9d700 1 -- 172.21.15.113:0/501533302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b04004090 msgr2=0x7f3b04004850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1ad9d700 1 --2- 172.21.15.113:0/501533302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b04004090 0x7f3b04004850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.182+0000 7f3b1ad9d700 1 -- 172.21.15.113:0/501533302 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b040035b0 con 0x7f3b04004d90 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.183+0000 7f3b1ad9d700 1 --2- 172.21.15.113:0/501533302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b04004d90 0x7f3b04005210 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400c1e0 tx=0x7f3b04005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b188f42a6697249 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.183+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14007b70 con 0x7f3b04004d90 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.183+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1400cd80 con 0x7f3b04004d90 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.183+0000 7f3b1959a700 1 -- 172.21.15.113:0/501533302 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14005ba0 con 0x7f3b04004d90 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.184+0000 7f3b20c19700 1 -- 172.21.15.113:0/501533302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b04004d90 msgr2=0x7f3b04005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.184+0000 7f3b20c19700 1 --2- 172.21.15.113:0/501533302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b04004d90 0x7f3b04005210 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400c1e0 tx=0x7f3b04005880 comp rx=0 tx=0).stop 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.184+0000 7f3b1ad9d700 1 -- 172.21.15.113:0/501533302 reap_dead start 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.184+0000 7f3b20c19700 1 -- 172.21.15.113:0/501533302 shutdown_connections 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.184+0000 7f3b20c19700 1 -- 172.21.15.113:0/501533302 >> 172.21.15.113:0/501533302 conn(0x7f3b1c06b620 msgr2=0x7f3b1c071610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.184+0000 7f3b20c19700 1 -- 172.21.15.113:0/501533302 shutdown_connections 2024-01-15T11:26:55.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.184+0000 7f3b20c19700 1 -- 172.21.15.113:0/501533302 wait complete. 2024-01-15T11:26:55.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.186+0000 7f3b20c19700 1 Processor -- start 2024-01-15T11:26:55.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.186+0000 7f3b20c19700 1 -- start start 2024-01-15T11:26:55.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.187+0000 7f3b20c19700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b04004d90 0x7f3b1c15c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:55.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.187+0000 7f3b20c19700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b1c0cd160 0x7f3b1c0c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:55.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.187+0000 7f3b20c19700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0cd5c0 0x7f3b1c0ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:55.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.187+0000 7f3b20c19700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0cc0c0 con 0x7f3b1c0cd160 2024-01-15T11:26:55.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.187+0000 7f3b20c19700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0cc3a0 con 0x7f3b04004d90 2024-01-15T11:26:55.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.187+0000 7f3b20c19700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0cc680 con 0x7f3b1c0cd5c0 2024-01-15T11:26:55.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.187+0000 7f3b19d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b1c0cd160 0x7f3b1c0c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:55.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.187+0000 7f3b19d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b1c0cd160 0x7f3b1c0c9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34890/0 (socket says 172.21.15.113:34890) 2024-01-15T11:26:55.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.187+0000 7f3b19d9b700 1 -- 172.21.15.113:0/1827012693 learned_addr learned my addr 172.21.15.113:0/1827012693 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b19d9b700 1 -- 172.21.15.113:0/1827012693 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0cd5c0 msgr2=0x7f3b1c0ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b1a59c700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b04004d90 0x7f3b1c15c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b1ad9d700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0cd5c0 0x7f3b1c0ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b19d9b700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0cd5c0 0x7f3b1c0ca060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b19d9b700 1 -- 172.21.15.113:0/1827012693 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b04004d90 msgr2=0x7f3b1c15c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b19d9b700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b04004d90 0x7f3b1c15c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b19d9b700 1 -- 172.21.15.113:0/1827012693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c0cc960 con 0x7f3b1c0cd160 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b1ad9d700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0cd5c0 0x7f3b1c0ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b1a59c700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b04004d90 0x7f3b1c15c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b19d9b700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b1c0cd160 0x7f3b1c0c9b20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000c490 tx=0x7f3b10001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b0affd700 1 -- 172.21.15.113:0/1827012693 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b100054e0 con 0x7f3b1c0cd160 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b0affd700 1 -- 172.21.15.113:0/1827012693 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b10003680 con 0x7f3b1c0cd160 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b20c19700 1 -- 172.21.15.113:0/1827012693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1c0ccc40 con 0x7f3b1c0cd160 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b0affd700 1 -- 172.21.15.113:0/1827012693 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b10003a30 con 0x7f3b1c0cd160 2024-01-15T11:26:55.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.188+0000 7f3b20c19700 1 -- 172.21.15.113:0/1827012693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1c0aabb0 con 0x7f3b1c0cd160 2024-01-15T11:26:55.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.190+0000 7f3b20c19700 1 -- 172.21.15.113:0/1827012693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1c0cce60 con 0x7f3b1c0cd160 2024-01-15T11:26:55.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.190+0000 7f3b0affd700 1 -- 172.21.15.113:0/1827012693 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f3b10003b90 con 0x7f3b1c0cd160 2024-01-15T11:26:55.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.190+0000 7f3b0affd700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3afc06b500 0x7f3afc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:55.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.191+0000 7f3b0affd700 1 -- 172.21.15.113:0/1827012693 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b100892b0 con 0x7f3b1c0cd160 2024-01-15T11:26:55.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.191+0000 7f3b1a59c700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3afc06b500 0x7f3afc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:55.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.191+0000 7f3b1a59c700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3afc06b500 0x7f3afc06d9c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0ca6e0 tx=0x7f3b0c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:55.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.197+0000 7f3b0affd700 1 -- 172.21.15.113:0/1827012693 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b10053f30 con 0x7f3b1c0cd160 2024-01-15T11:26:55.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.601+0000 7f3b20c19700 1 -- 172.21.15.113:0/1827012693 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b1c0cf730 con 0x7f3afc06b500 2024-01-15T11:26:55.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.617+0000 7f3b0affd700 1 -- 172.21.15.113:0/1827012693 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f3b1c0cf730 con 0x7f3afc06b500 2024-01-15T11:26:55.617 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:26:55.617 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-15T11:26:55.617 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-15T11:26:55.617 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-15T11:26:55.617 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-15T11:26:55.617 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-15T11:26:55.617 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-15T11:26:55.617 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:55.617 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-15T11:26:55.617 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 33.33333333333333, 2024-01-15T11:26:55.618 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T11:26:55.618 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "1/3", 2024-01-15T11:26:55.618 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-15T11:26:55.618 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-15T11:26:55.618 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:55.618 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:55.618 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:26:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.628+0000 7f3b20c19700 1 -- 172.21.15.113:0/1827012693 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3afc06b500 msgr2=0x7f3afc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.628+0000 7f3b20c19700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3afc06b500 0x7f3afc06d9c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0ca6e0 tx=0x7f3b0c00b040 comp rx=0 tx=0).stop 2024-01-15T11:26:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.628+0000 7f3b20c19700 1 -- 172.21.15.113:0/1827012693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b1c0cd160 msgr2=0x7f3b1c0c9b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.628+0000 7f3b20c19700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b1c0cd160 0x7f3b1c0c9b20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000c490 tx=0x7f3b10001e70 comp rx=0 tx=0).stop 2024-01-15T11:26:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.630+0000 7f3b20c19700 1 -- 172.21.15.113:0/1827012693 shutdown_connections 2024-01-15T11:26:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.630+0000 7f3b20c19700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f3afc06b500 0x7f3afc06d9c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.630+0000 7f3b20c19700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f3b1c0cd5c0 0x7f3b1c0ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.630+0000 7f3b20c19700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3b04004d90 0x7f3b1c15c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.630+0000 7f3b20c19700 1 --2- 172.21.15.113:0/1827012693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b1c0cd160 0x7f3b1c0c9b20 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:55.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.630+0000 7f3b20c19700 1 -- 172.21.15.113:0/1827012693 >> 172.21.15.113:0/1827012693 conn(0x7f3b1c06b620 msgr2=0x7f3b1c0d2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:55.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.630+0000 7f3b20c19700 1 -- 172.21.15.113:0/1827012693 shutdown_connections 2024-01-15T11:26:55.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:55.630+0000 7f3b20c19700 1 -- 172.21.15.113:0/1827012693 wait complete. 2024-01-15T11:26:55.660 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T11:26:56.661 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T11:26:56.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.792+0000 7f15a91e9700 1 Processor -- start 2024-01-15T11:26:56.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.792+0000 7f15a91e9700 1 -- start start 2024-01-15T11:26:56.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.792+0000 7f15a91e9700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f15a409b110 0x7f15a40d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:56.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.792+0000 7f15a91e9700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40d0a70 con 0x7f15a4161a70 2024-01-15T11:26:56.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.792+0000 7f15a91e9700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40d1c10 con 0x7f15a4161690 2024-01-15T11:26:56.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.792+0000 7f15a91e9700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a40b2ab0 con 0x7f15a409b110 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.793+0000 7f15a359e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f15a409b110 0x7f15a40d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.793+0000 7f15a259c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f15a4161a70 0x7f15a40cc4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51920/0 (socket says 172.21.15.113:51920) 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.793+0000 7f15a259c700 1 -- 172.21.15.113:0/2918257950 learned_addr learned my addr 172.21.15.113:0/2918257950 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.793+0000 7f15a359e700 1 -- 172.21.15.113:0/2918257950 >> v1:172.21.15.141:6789/0 conn(0x7f15a4161690 legacy=0x7f15a409aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.793+0000 7f15a359e700 1 -- 172.21.15.113:0/2918257950 >> v1:172.21.15.113:6789/0 conn(0x7f15a4161a70 legacy=0x7f15a40cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.793+0000 7f15a359e700 1 -- 172.21.15.113:0/2918257950 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40b2d90 con 0x7f15a409b110 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.793+0000 7f15a1d9b700 1 -- 172.21.15.113:0/2918257950 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520159598 0 0) 0x7f15a40d0a70 con 0x7f15a4161a70 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.793+0000 7f15a359e700 1 --2- 172.21.15.113:0/2918257950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a409b110 0x7f15a40d0f20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f159c009d90 tx=0x7f159c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99153c171de5cddd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.793+0000 7f15a1d9b700 1 -- 172.21.15.113:0/2918257950 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c00e070 con 0x7f15a409b110 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.793+0000 7f15a1d9b700 1 -- 172.21.15.113:0/2918257950 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c00bb90 con 0x7f15a409b110 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.793+0000 7f15a1d9b700 1 -- 172.21.15.113:0/2918257950 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c003a20 con 0x7f15a409b110 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.794+0000 7f15a91e9700 1 -- 172.21.15.113:0/2918257950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a409b110 msgr2=0x7f15a40d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.794+0000 7f15a91e9700 1 --2- 172.21.15.113:0/2918257950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a409b110 0x7f15a40d0f20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f159c009d90 tx=0x7f159c00b330 comp rx=0 tx=0).stop 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.794+0000 7f15a91e9700 1 -- 172.21.15.113:0/2918257950 shutdown_connections 2024-01-15T11:26:56.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.794+0000 7f15a91e9700 1 --2- 172.21.15.113:0/2918257950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a409b110 0x7f15a40d0f20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:56.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.794+0000 7f15a91e9700 1 -- 172.21.15.113:0/2918257950 >> 172.21.15.113:0/2918257950 conn(0x7f15a40d1e50 msgr2=0x7f15a40d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:56.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.794+0000 7f15a91e9700 1 -- 172.21.15.113:0/2918257950 shutdown_connections 2024-01-15T11:26:56.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.794+0000 7f15a91e9700 1 -- 172.21.15.113:0/2918257950 wait complete. 2024-01-15T11:26:56.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.795+0000 7f15a91e9700 1 Processor -- start 2024-01-15T11:26:56.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.795+0000 7f15a91e9700 1 -- start start 2024-01-15T11:26:56.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a91e9700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15a409b110 0x7f15a40c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:56.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a91e9700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f15a4161690 0x7f15a4152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:56.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a91e9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a4161a70 0x7f15a40b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:56.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a91e9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40c5a20 con 0x7f15a4161a70 2024-01-15T11:26:56.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a91e9700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40b6c60 con 0x7f15a409b110 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a91e9700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40c8180 con 0x7f15a4161690 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a359e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a4161a70 0x7f15a40b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a359e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a4161a70 0x7f15a40b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34912/0 (socket says 172.21.15.113:34912) 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a359e700 1 -- 172.21.15.113:0/2247892518 learned_addr learned my addr 172.21.15.113:0/2247892518 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a2d9d700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15a409b110 0x7f15a40c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a259c700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f15a4161690 0x7f15a4152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a359e700 1 -- 172.21.15.113:0/2247892518 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f15a4161690 msgr2=0x7f15a4152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a359e700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f15a4161690 0x7f15a4152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a359e700 1 -- 172.21.15.113:0/2247892518 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15a409b110 msgr2=0x7f15a40c7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a359e700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15a409b110 0x7f15a40c7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a359e700 1 -- 172.21.15.113:0/2247892518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c009a70 con 0x7f15a4161a70 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a259c700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f15a4161690 0x7f15a4152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.796+0000 7f15a359e700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a4161a70 0x7f15a40b4af0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f15a4072870 tx=0x7f159c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.797+0000 7f158b7fe700 1 -- 172.21.15.113:0/2247892518 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c012630 con 0x7f15a4161a70 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.797+0000 7f158b7fe700 1 -- 172.21.15.113:0/2247892518 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c012790 con 0x7f15a4161a70 2024-01-15T11:26:56.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.797+0000 7f15a91e9700 1 -- 172.21.15.113:0/2247892518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a40b6f20 con 0x7f15a4161a70 2024-01-15T11:26:56.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.797+0000 7f158b7fe700 1 -- 172.21.15.113:0/2247892518 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c012b40 con 0x7f15a4161a70 2024-01-15T11:26:56.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.797+0000 7f15a91e9700 1 -- 172.21.15.113:0/2247892518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a40af760 con 0x7f15a4161a70 2024-01-15T11:26:56.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.798+0000 7f15a91e9700 1 -- 172.21.15.113:0/2247892518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a40c5f00 con 0x7f15a4161a70 2024-01-15T11:26:56.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.798+0000 7f158b7fe700 1 -- 172.21.15.113:0/2247892518 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f159c01d420 con 0x7f15a4161a70 2024-01-15T11:26:56.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.799+0000 7f158b7fe700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f159006b500 0x7f159006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:56.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.799+0000 7f15a2d9d700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f159006b500 0x7f159006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:56.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.799+0000 7f158b7fe700 1 -- 172.21.15.113:0/2247892518 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f159c08bbb0 con 0x7f15a4161a70 2024-01-15T11:26:56.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.799+0000 7f15a2d9d700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f159006b500 0x7f159006d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1594001730 tx=0x7f1594009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:56.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:56.803+0000 7f158b7fe700 1 -- 172.21.15.113:0/2247892518 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15a40c5f00 con 0x7f15a4161a70 2024-01-15T11:26:57.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.036+0000 7f15a91e9700 1 -- 172.21.15.113:0/2247892518 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f15a40c5f00 con 0x7f159006b500 2024-01-15T11:26:57.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.041+0000 7f158b7fe700 1 -- 172.21.15.113:0/2247892518 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f15a40c5f00 con 0x7f159006b500 2024-01-15T11:26:57.041 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:26:57.041 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-15T11:26:57.041 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-15T11:26:57.041 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-15T11:26:57.041 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-15T11:26:57.042 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-15T11:26:57.042 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-15T11:26:57.042 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:57.042 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-15T11:26:57.042 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 66.66666666666666, 2024-01-15T11:26:57.042 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T11:26:57.042 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "2/3", 2024-01-15T11:26:57.042 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-15T11:26:57.042 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-15T11:26:57.042 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:57.042 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:57.042 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:26:57.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.044+0000 7f15a91e9700 1 -- 172.21.15.113:0/2247892518 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f159006b500 msgr2=0x7f159006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:57.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.044+0000 7f15a91e9700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f159006b500 0x7f159006d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1594001730 tx=0x7f1594009040 comp rx=0 tx=0).stop 2024-01-15T11:26:57.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.044+0000 7f15a91e9700 1 -- 172.21.15.113:0/2247892518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a4161a70 msgr2=0x7f15a40b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:57.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.044+0000 7f15a91e9700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a4161a70 0x7f15a40b4af0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f15a4072870 tx=0x7f159c004ad0 comp rx=0 tx=0).stop 2024-01-15T11:26:57.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.044+0000 7f15a91e9700 1 -- 172.21.15.113:0/2247892518 shutdown_connections 2024-01-15T11:26:57.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.044+0000 7f15a91e9700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f159006b500 0x7f159006d9c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:57.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.044+0000 7f15a91e9700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a4161a70 0x7f15a40b4af0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:57.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.044+0000 7f15a91e9700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f15a409b110 0x7f15a40c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:57.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.044+0000 7f15a91e9700 1 --2- 172.21.15.113:0/2247892518 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f15a4161690 0x7f15a4152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:57.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.044+0000 7f15a91e9700 1 -- 172.21.15.113:0/2247892518 >> 172.21.15.113:0/2247892518 conn(0x7f15a40d1e50 msgr2=0x7f15a415fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:57.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.044+0000 7f15a91e9700 1 -- 172.21.15.113:0/2247892518 shutdown_connections 2024-01-15T11:26:57.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:57.044+0000 7f15a91e9700 1 -- 172.21.15.113:0/2247892518 wait complete. 2024-01-15T11:26:57.055 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T11:26:58.056 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T11:26:58.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.300+0000 7fa415975700 1 Processor -- start 2024-01-15T11:26:58.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.301+0000 7fa415975700 1 -- start start 2024-01-15T11:26:58.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.302+0000 7fa415975700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa41015e110 0x7fa41006f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:58.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.302+0000 7fa415975700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa41006e420 0x7fa41006e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:58.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.302+0000 7fa415975700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4100affe0 con 0x7fa4100afb30 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.302+0000 7fa415975700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41006d980 con 0x7fa41015e110 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.302+0000 7fa415975700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4100b0260 con 0x7fa41006e420 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.302+0000 7fa40effd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa41015e110 0x7fa41006f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.302+0000 7fa40effd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa41015e110 0x7fa41006f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34922/0 (socket says 172.21.15.113:34922) 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.302+0000 7fa40effd700 1 -- 172.21.15.113:0/1763060353 learned_addr learned my addr 172.21.15.113:0/1763060353 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.303+0000 7fa40effd700 1 -- 172.21.15.113:0/1763060353 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa41006e420 msgr2=0x7fa41006e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.303+0000 7fa40effd700 1 --2- 172.21.15.113:0/1763060353 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa41006e420 0x7fa41006e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.303+0000 7fa40effd700 1 -- 172.21.15.113:0/1763060353 >> v1:172.21.15.113:6789/0 conn(0x7fa4100afb30 legacy=0x7fa41006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.303+0000 7fa40effd700 1 -- 172.21.15.113:0/1763060353 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4100b0540 con 0x7fa41015e110 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.303+0000 7fa40effd700 1 --2- 172.21.15.113:0/1763060353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa41015e110 0x7fa41006f680 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa4000098f0 tx=0x7fa400009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e33c14676ee603 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.303+0000 7fa40dffb700 1 -- 172.21.15.113:0/1763060353 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40000b8a0 con 0x7fa41015e110 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.303+0000 7fa40dffb700 1 -- 172.21.15.113:0/1763060353 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40000ba00 con 0x7fa41015e110 2024-01-15T11:26:58.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.303+0000 7fa40dffb700 1 -- 172.21.15.113:0/1763060353 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40000bd90 con 0x7fa41015e110 2024-01-15T11:26:58.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.304+0000 7fa415975700 1 -- 172.21.15.113:0/1763060353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa41015e110 msgr2=0x7fa41006f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:58.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.304+0000 7fa415975700 1 --2- 172.21.15.113:0/1763060353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa41015e110 0x7fa41006f680 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa4000098f0 tx=0x7fa400009c00 comp rx=0 tx=0).stop 2024-01-15T11:26:58.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.304+0000 7fa415975700 1 -- 172.21.15.113:0/1763060353 shutdown_connections 2024-01-15T11:26:58.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.304+0000 7fa415975700 1 --2- 172.21.15.113:0/1763060353 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa41006e420 0x7fa41006e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:58.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.304+0000 7fa415975700 1 --2- 172.21.15.113:0/1763060353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa41015e110 0x7fa41006f680 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:58.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.304+0000 7fa415975700 1 -- 172.21.15.113:0/1763060353 >> 172.21.15.113:0/1763060353 conn(0x7fa4100b3250 msgr2=0x7fa4100d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:58.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.304+0000 7fa415975700 1 -- 172.21.15.113:0/1763060353 shutdown_connections 2024-01-15T11:26:58.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.304+0000 7fa415975700 1 -- 172.21.15.113:0/1763060353 wait complete. 2024-01-15T11:26:58.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.306+0000 7fa415975700 1 Processor -- start 2024-01-15T11:26:58.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.306+0000 7fa415975700 1 -- start start 2024-01-15T11:26:58.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.307+0000 7fa415975700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa41006e420 0x7fa410146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:58.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.307+0000 7fa415975700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4100afb30 0x7fa4100cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:58.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.307+0000 7fa415975700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa41015e110 0x7fa41009e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:58.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.307+0000 7fa415975700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa410146bc0 con 0x7fa4100afb30 2024-01-15T11:26:58.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.307+0000 7fa415975700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4100d04b0 con 0x7fa41015e110 2024-01-15T11:26:58.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.307+0000 7fa415975700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4100cf9e0 con 0x7fa41006e420 2024-01-15T11:26:58.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.307+0000 7fa40e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4100afb30 0x7fa4100cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:58.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.307+0000 7fa40e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4100afb30 0x7fa4100cf4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34928/0 (socket says 172.21.15.113:34928) 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.307+0000 7fa40e7fc700 1 -- 172.21.15.113:0/3268042334 learned_addr learned my addr 172.21.15.113:0/3268042334 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.307+0000 7fa40effd700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa41006e420 0x7fa410146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.307+0000 7fa40f7fe700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa41015e110 0x7fa41009e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.308+0000 7fa40e7fc700 1 -- 172.21.15.113:0/3268042334 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa41006e420 msgr2=0x7fa410146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.308+0000 7fa40e7fc700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa41006e420 0x7fa410146680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.308+0000 7fa40e7fc700 1 -- 172.21.15.113:0/3268042334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa41015e110 msgr2=0x7fa41009e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.308+0000 7fa40e7fc700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa41015e110 0x7fa41009e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.308+0000 7fa40e7fc700 1 -- 172.21.15.113:0/3268042334 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa400009550 con 0x7fa4100afb30 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.308+0000 7fa40f7fe700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa41015e110 0x7fa41009e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.308+0000 7fa40e7fc700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4100afb30 0x7fa4100cf4a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa40400c510 tx=0x7fa40400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.308+0000 7fa3f7fff700 1 -- 172.21.15.113:0/3268042334 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4040052d0 con 0x7fa4100afb30 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.308+0000 7fa3f7fff700 1 -- 172.21.15.113:0/3268042334 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa404005430 con 0x7fa4100afb30 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.308+0000 7fa415975700 1 -- 172.21.15.113:0/3268042334 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41006b880 con 0x7fa4100afb30 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.308+0000 7fa3f7fff700 1 -- 172.21.15.113:0/3268042334 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4040038c0 con 0x7fa4100afb30 2024-01-15T11:26:58.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.308+0000 7fa415975700 1 -- 172.21.15.113:0/3268042334 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41009aba0 con 0x7fa4100afb30 2024-01-15T11:26:58.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.310+0000 7fa3f7fff700 1 -- 172.21.15.113:0/3268042334 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fa404003a20 con 0x7fa4100afb30 2024-01-15T11:26:58.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.311+0000 7fa3f7fff700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa3fc06b550 0x7fa3fc06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:58.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.311+0000 7fa3f7fff700 1 -- 172.21.15.113:0/3268042334 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4040891c0 con 0x7fa4100afb30 2024-01-15T11:26:58.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.312+0000 7fa415975700 1 -- 172.21.15.113:0/3268042334 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f00052f0 con 0x7fa4100afb30 2024-01-15T11:26:58.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.313+0000 7fa40effd700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa3fc06b550 0x7fa3fc06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:58.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.314+0000 7fa40effd700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa3fc06b550 0x7fa3fc06da10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa400014040 tx=0x7fa400008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:58.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.317+0000 7fa3f7fff700 1 -- 172.21.15.113:0/3268042334 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa404053f20 con 0x7fa4100afb30 2024-01-15T11:26:58.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.695+0000 7fa415975700 1 -- 172.21.15.113:0/3268042334 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3f0000bc0 con 0x7fa3fc06b550 2024-01-15T11:26:58.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.712+0000 7fa3f7fff700 1 -- 172.21.15.113:0/3268042334 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fa3f0000bc0 con 0x7fa3fc06b550 2024-01-15T11:26:58.711 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 66.66666666666666, 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "2/3", 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-15T11:26:58.712 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:58.713 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:26:58.713 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:26:58.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.716+0000 7fa415975700 1 -- 172.21.15.113:0/3268042334 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa3fc06b550 msgr2=0x7fa3fc06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:58.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.716+0000 7fa415975700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa3fc06b550 0x7fa3fc06da10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa400014040 tx=0x7fa400008030 comp rx=0 tx=0).stop 2024-01-15T11:26:58.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.716+0000 7fa415975700 1 -- 172.21.15.113:0/3268042334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4100afb30 msgr2=0x7fa4100cf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:58.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.716+0000 7fa415975700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4100afb30 0x7fa4100cf4a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa40400c510 tx=0x7fa40400c820 comp rx=0 tx=0).stop 2024-01-15T11:26:58.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.717+0000 7fa415975700 1 -- 172.21.15.113:0/3268042334 shutdown_connections 2024-01-15T11:26:58.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.717+0000 7fa415975700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fa3fc06b550 0x7fa3fc06da10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:58.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.717+0000 7fa415975700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa41015e110 0x7fa41009e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:58.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.717+0000 7fa415975700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fa41006e420 0x7fa410146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:58.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.717+0000 7fa415975700 1 --2- 172.21.15.113:0/3268042334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4100afb30 0x7fa4100cf4a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:58.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.717+0000 7fa415975700 1 -- 172.21.15.113:0/3268042334 >> 172.21.15.113:0/3268042334 conn(0x7fa4100b3250 msgr2=0x7fa4100d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:58.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.717+0000 7fa415975700 1 -- 172.21.15.113:0/3268042334 shutdown_connections 2024-01-15T11:26:58.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:58.717+0000 7fa415975700 1 -- 172.21.15.113:0/3268042334 wait complete. 2024-01-15T11:26:58.741 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T11:26:59.741 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T11:26:59.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.866+0000 7fcad2798700 1 Processor -- start 2024-01-15T11:26:59.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.866+0000 7fcad2798700 1 -- start start 2024-01-15T11:26:59.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.866+0000 7fcad2798700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcacc0b2b00 0x7fcacc0d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.866+0000 7fcad2798700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcacc162920 con 0x7fcacc15e1e0 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.866+0000 7fcad2798700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcacc0ce940 con 0x7fcacc0d1390 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.866+0000 7fcad2798700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcacc0ce730 con 0x7fcacc0b2b00 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcad0d35700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcacc0b2b00 0x7fcacc0d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcacbfff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fcacc15e1e0 0x7fcacc0d0c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51928/0 (socket says 172.21.15.113:51928) 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcad0d35700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcacc0b2b00 0x7fcacc0d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34940/0 (socket says 172.21.15.113:34940) 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcacbfff700 1 -- 172.21.15.113:0/1874853734 learned_addr learned my addr 172.21.15.113:0/1874853734 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcacaffd700 1 -- 172.21.15.113:0/1874853734 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475609934 0 0) 0x7fcacc162920 con 0x7fcacc15e1e0 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcacaffd700 1 -- 172.21.15.113:0/1874853734 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcab40035b0 con 0x7fcacc15e1e0 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcad0d35700 1 -- 172.21.15.113:0/1874853734 >> v1:172.21.15.141:6789/0 conn(0x7fcacc0d1390 legacy=0x7fcacc0d20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcad0d35700 1 -- 172.21.15.113:0/1874853734 >> v1:172.21.15.113:6789/0 conn(0x7fcacc15e1e0 legacy=0x7fcacc0d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcad0d35700 1 -- 172.21.15.113:0/1874853734 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcacc0cec20 con 0x7fcacc0b2b00 2024-01-15T11:26:59.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcad0d35700 1 --2- 172.21.15.113:0/1874853734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcacc0b2b00 0x7fcacc0d27b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009d90 tx=0x7fcac400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e27161897c6101 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:59.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcacaffd700 1 -- 172.21.15.113:0/1874853734 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac400e070 con 0x7fcacc0b2b00 2024-01-15T11:26:59.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcacaffd700 1 -- 172.21.15.113:0/1874853734 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac400bb20 con 0x7fcacc0b2b00 2024-01-15T11:26:59.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.867+0000 7fcacaffd700 1 -- 172.21.15.113:0/1874853734 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4012430 con 0x7fcacc0b2b00 2024-01-15T11:26:59.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.868+0000 7fcad2798700 1 -- 172.21.15.113:0/1874853734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcacc0b2b00 msgr2=0x7fcacc0d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:59.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.868+0000 7fcad2798700 1 --2- 172.21.15.113:0/1874853734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcacc0b2b00 0x7fcacc0d27b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009d90 tx=0x7fcac400b330 comp rx=0 tx=0).stop 2024-01-15T11:26:59.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.868+0000 7fcad2798700 1 -- 172.21.15.113:0/1874853734 shutdown_connections 2024-01-15T11:26:59.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.868+0000 7fcad2798700 1 --2- 172.21.15.113:0/1874853734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcacc0b2b00 0x7fcacc0d27b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:59.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.868+0000 7fcad2798700 1 -- 172.21.15.113:0/1874853734 >> 172.21.15.113:0/1874853734 conn(0x7fcacc0717f0 msgr2=0x7fcacc071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:26:59.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.868+0000 7fcad2798700 1 -- 172.21.15.113:0/1874853734 shutdown_connections 2024-01-15T11:26:59.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.868+0000 7fcad2798700 1 -- 172.21.15.113:0/1874853734 wait complete. 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.869+0000 7fcad2798700 1 Processor -- start 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.869+0000 7fcad2798700 1 -- start start 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcad2798700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcacc0b2b00 0x7fcacc1f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcad2798700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcacc0d1390 0x7fcacc0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcad2798700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcacc15e1e0 0x7fcacc0c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcad2798700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc1f3810 con 0x7fcacc0d1390 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcad2798700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc0ccfd0 con 0x7fcacc15e1e0 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcad2798700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc0cd2b0 con 0x7fcacc0b2b00 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcacb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcacc0d1390 0x7fcacc0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcacb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcacc0d1390 0x7fcacc0cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34948/0 (socket says 172.21.15.113:34948) 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcacb7fe700 1 -- 172.21.15.113:0/2082209835 learned_addr learned my addr 172.21.15.113:0/2082209835 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcad0d35700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcacc15e1e0 0x7fcacc0c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcacbfff700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcacc0b2b00 0x7fcacc1f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcacb7fe700 1 -- 172.21.15.113:0/2082209835 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcacc0b2b00 msgr2=0x7fcacc1f32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcacb7fe700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcacc0b2b00 0x7fcacc1f32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcacb7fe700 1 -- 172.21.15.113:0/2082209835 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcacc15e1e0 msgr2=0x7fcacc0c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcacb7fe700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcacc15e1e0 0x7fcacc0c9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcacb7fe700 1 -- 172.21.15.113:0/2082209835 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac4009a70 con 0x7fcacc0d1390 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcad0d35700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcacc15e1e0 0x7fcacc0c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcacbfff700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcacc0b2b00 0x7fcacc1f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcacb7fe700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcacc0d1390 0x7fcacc0cc5f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcac000c4e0 tx=0x7fcac000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.870+0000 7fcac8ff9700 1 -- 172.21.15.113:0/2082209835 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00052d0 con 0x7fcacc0d1390 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.871+0000 7fcac8ff9700 1 -- 172.21.15.113:0/2082209835 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac0005430 con 0x7fcacc0d1390 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.871+0000 7fcad2798700 1 -- 172.21.15.113:0/2082209835 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcacc0cf670 con 0x7fcacc0d1390 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.871+0000 7fcac8ff9700 1 -- 172.21.15.113:0/2082209835 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00038c0 con 0x7fcacc0d1390 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.871+0000 7fcad2798700 1 -- 172.21.15.113:0/2082209835 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcacc0cd870 con 0x7fcacc0d1390 2024-01-15T11:26:59.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.872+0000 7fcad2798700 1 -- 172.21.15.113:0/2082209835 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcacc069020 con 0x7fcacc0d1390 2024-01-15T11:26:59.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.872+0000 7fcac8ff9700 1 -- 172.21.15.113:0/2082209835 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fcac0003a20 con 0x7fcacc0d1390 2024-01-15T11:26:59.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.872+0000 7fcac8ff9700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fcab806b500 0x7fcab806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:26:59.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.873+0000 7fcacbfff700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fcab806b500 0x7fcab806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:26:59.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.873+0000 7fcac8ff9700 1 -- 172.21.15.113:0/2082209835 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcac0089380 con 0x7fcacc0d1390 2024-01-15T11:26:59.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.873+0000 7fcacbfff700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fcab806b500 0x7fcab806d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcabc0011e0 tx=0x7fcabc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:26:59.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:26:59.877+0000 7fcac8ff9700 1 -- 172.21.15.113:0/2082209835 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcac0053f70 con 0x7fcacc0d1390 2024-01-15T11:27:00.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.108+0000 7fcad2798700 1 -- 172.21.15.113:0/2082209835 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcacc0cf4f0 con 0x7fcab806b500 2024-01-15T11:27:00.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.716+0000 7fcac8ff9700 1 -- 172.21.15.113:0/2082209835 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fcabc001470 con 0x7fcab806b500 2024-01-15T11:27:00.716 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-15T11:27:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-15T11:27:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "complete" 2024-01-15T11:27:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:27:00.716 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-15T11:27:00.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.719+0000 7fcad2798700 1 -- 172.21.15.113:0/2082209835 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fcab806b500 msgr2=0x7fcab806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:00.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.719+0000 7fcad2798700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fcab806b500 0x7fcab806d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcabc0011e0 tx=0x7fcabc009040 comp rx=0 tx=0).stop 2024-01-15T11:27:00.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.719+0000 7fcad2798700 1 -- 172.21.15.113:0/2082209835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcacc0d1390 msgr2=0x7fcacc0cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:00.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.719+0000 7fcad2798700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcacc0d1390 0x7fcacc0cc5f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcac000c4e0 tx=0x7fcac000c8a0 comp rx=0 tx=0).stop 2024-01-15T11:27:00.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.719+0000 7fcad2798700 1 -- 172.21.15.113:0/2082209835 shutdown_connections 2024-01-15T11:27:00.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.719+0000 7fcad2798700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fcab806b500 0x7fcab806d9c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:00.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.719+0000 7fcad2798700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcacc15e1e0 0x7fcacc0c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:00.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.719+0000 7fcad2798700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcacc0b2b00 0x7fcacc1f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:00.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.719+0000 7fcad2798700 1 --2- 172.21.15.113:0/2082209835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcacc0d1390 0x7fcacc0cc5f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:00.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.719+0000 7fcad2798700 1 -- 172.21.15.113:0/2082209835 >> 172.21.15.113:0/2082209835 conn(0x7fcacc0717f0 msgr2=0x7fcacc0d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:00.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.719+0000 7fcad2798700 1 -- 172.21.15.113:0/2082209835 shutdown_connections 2024-01-15T11:27:00.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.719+0000 7fcad2798700 1 -- 172.21.15.113:0/2082209835 wait complete. 2024-01-15T11:27:00.733 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T11:27:00.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.862+0000 7fb1c17af700 1 Processor -- start 2024-01-15T11:27:00.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.862+0000 7fb1c17af700 1 -- start start 2024-01-15T11:27:00.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1c17af700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb1bc15e1e0 0x7fb1bc0d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:00.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1c17af700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb1bc0d3a00 0x7fb1bc0d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:00.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1c17af700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1bc0d20b0 con 0x7fb1bc0d0db0 2024-01-15T11:27:00.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1c17af700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1bc0d3150 con 0x7fb1bc0d3a00 2024-01-15T11:27:00.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1c17af700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc0d1240 con 0x7fb1bc15e1e0 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1ba7fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb1bc0d3a00 0x7fb1bc0d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1ba7fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb1bc0d3a00 0x7fb1bc0d0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34964/0 (socket says 172.21.15.113:34964) 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1ba7fc700 1 -- 172.21.15.113:0/2395886710 learned_addr learned my addr 172.21.15.113:0/2395886710 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1baffd700 1 --2- 172.21.15.113:0/2395886710 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb1bc15e1e0 0x7fb1bc0d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1ba7fc700 1 -- 172.21.15.113:0/2395886710 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb1bc15e1e0 msgr2=0x7fb1bc0d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1ba7fc700 1 --2- 172.21.15.113:0/2395886710 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb1bc15e1e0 0x7fb1bc0d34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1ba7fc700 1 -- 172.21.15.113:0/2395886710 >> v1:172.21.15.141:6789/0 conn(0x7fb1bc0d0db0 legacy=0x7fb1bc06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1ba7fc700 1 -- 172.21.15.113:0/2395886710 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1bc0d2390 con 0x7fb1bc0d3a00 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1baffd700 1 --2- 172.21.15.113:0/2395886710 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb1bc15e1e0 0x7fb1bc0d34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.863+0000 7fb1ba7fc700 1 --2- 172.21.15.113:0/2395886710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1bc0d3a00 0x7fb1bc0d0870 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb1b00098f0 tx=0x7fb1b0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d695270c5b5b61 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.864+0000 7fb1b9ffb700 1 -- 172.21.15.113:0/2395886710 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0004230 con 0x7fb1bc0d3a00 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.864+0000 7fb1b9ffb700 1 -- 172.21.15.113:0/2395886710 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b0004390 con 0x7fb1bc0d3a00 2024-01-15T11:27:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.864+0000 7fb1b9ffb700 1 -- 172.21.15.113:0/2395886710 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b00036a0 con 0x7fb1bc0d3a00 2024-01-15T11:27:00.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.864+0000 7fb1c17af700 1 -- 172.21.15.113:0/2395886710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1bc0d3a00 msgr2=0x7fb1bc0d0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:00.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.864+0000 7fb1c17af700 1 --2- 172.21.15.113:0/2395886710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1bc0d3a00 0x7fb1bc0d0870 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb1b00098f0 tx=0x7fb1b0009c00 comp rx=0 tx=0).stop 2024-01-15T11:27:00.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.864+0000 7fb1c17af700 1 -- 172.21.15.113:0/2395886710 shutdown_connections 2024-01-15T11:27:00.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.864+0000 7fb1c17af700 1 --2- 172.21.15.113:0/2395886710 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb1bc15e1e0 0x7fb1bc0d34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:00.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.864+0000 7fb1c17af700 1 --2- 172.21.15.113:0/2395886710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1bc0d3a00 0x7fb1bc0d0870 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:00.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.864+0000 7fb1c17af700 1 -- 172.21.15.113:0/2395886710 >> 172.21.15.113:0/2395886710 conn(0x7fb1bc0b5b50 msgr2=0x7fb1bc0b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:00.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.864+0000 7fb1c17af700 1 -- 172.21.15.113:0/2395886710 shutdown_connections 2024-01-15T11:27:00.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.864+0000 7fb1c17af700 1 -- 172.21.15.113:0/2395886710 wait complete. 2024-01-15T11:27:00.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.865+0000 7fb1c17af700 1 Processor -- start 2024-01-15T11:27:00.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.866+0000 7fb1c17af700 1 -- start start 2024-01-15T11:27:00.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.866+0000 7fb1c17af700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1bc0d0db0 0x7fb1bc152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:00.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.866+0000 7fb1c17af700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb1bc0d3a00 0x7fb1bc1555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:00.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.866+0000 7fb1c17af700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1bc15e1e0 0x7fb1bc155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:00.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.866+0000 7fb1c17af700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc0cafd0 con 0x7fb1bc0d0db0 2024-01-15T11:27:00.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.866+0000 7fb1c17af700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc152fa0 con 0x7fb1bc15e1e0 2024-01-15T11:27:00.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.866+0000 7fb1c17af700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc156110 con 0x7fb1bc0d3a00 2024-01-15T11:27:00.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.866+0000 7fb1baffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1bc0d0db0 0x7fb1bc152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.866+0000 7fb1baffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1bc0d0db0 0x7fb1bc152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34972/0 (socket says 172.21.15.113:34972) 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.866+0000 7fb1baffd700 1 -- 172.21.15.113:0/398427316 learned_addr learned my addr 172.21.15.113:0/398427316 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1ba7fc700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb1bc0d3a00 0x7fb1bc1555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1bb7fe700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1bc15e1e0 0x7fb1bc155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1baffd700 1 -- 172.21.15.113:0/398427316 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb1bc0d3a00 msgr2=0x7fb1bc1555a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1baffd700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb1bc0d3a00 0x7fb1bc1555a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1baffd700 1 -- 172.21.15.113:0/398427316 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1bc15e1e0 msgr2=0x7fb1bc155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1baffd700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1bc15e1e0 0x7fb1bc155ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1baffd700 1 -- 172.21.15.113:0/398427316 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b0009550 con 0x7fb1bc0d0db0 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1ba7fc700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb1bc0d3a00 0x7fb1bc1555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1baffd700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1bc0d0db0 0x7fb1bc152470 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac00c7f0 tx=0x7fb1ac00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1a37fe700 1 -- 172.21.15.113:0/398427316 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac011800 con 0x7fb1bc0d0db0 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1a37fe700 1 -- 172.21.15.113:0/398427316 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ac011960 con 0x7fb1bc0d0db0 2024-01-15T11:27:00.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1c17af700 1 -- 172.21.15.113:0/398427316 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1bc0cb240 con 0x7fb1bc0d0db0 2024-01-15T11:27:00.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1a37fe700 1 -- 172.21.15.113:0/398427316 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac011d10 con 0x7fb1bc0d0db0 2024-01-15T11:27:00.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.867+0000 7fb1c17af700 1 -- 172.21.15.113:0/398427316 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1bc1529b0 con 0x7fb1bc0d0db0 2024-01-15T11:27:00.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.868+0000 7fb1c17af700 1 -- 172.21.15.113:0/398427316 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1bc0751e0 con 0x7fb1bc0d0db0 2024-01-15T11:27:00.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.868+0000 7fb1a37fe700 1 -- 172.21.15.113:0/398427316 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fb1ac018360 con 0x7fb1bc0d0db0 2024-01-15T11:27:00.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.869+0000 7fb1a37fe700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb1a806b500 0x7fb1a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:00.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.869+0000 7fb1a37fe700 1 -- 172.21.15.113:0/398427316 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1ac089f70 con 0x7fb1bc0d0db0 2024-01-15T11:27:00.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.869+0000 7fb1ba7fc700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb1a806b500 0x7fb1a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:00.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.869+0000 7fb1ba7fc700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb1a806b500 0x7fb1a806d9c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009130 tx=0x7fb1b0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:00.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:00.873+0000 7fb1a37fe700 1 -- 172.21.15.113:0/398427316 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1ac0bce90 con 0x7fb1bc0d0db0 2024-01-15T11:27:01.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.092+0000 7fb1c17af700 1 -- 172.21.15.113:0/398427316 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1bc0d2950 con 0x7fb1a806b500 2024-01-15T11:27:01.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.093+0000 7fb1a37fe700 1 -- 172.21.15.113:0/398427316 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb1b0003720 con 0x7fb1a806b500 2024-01-15T11:27:01.093 INFO:teuthology.orchestra.run.smithi113.stdout:[ 2024-01-15T11:27:01.093 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:27:01.093 INFO:teuthology.orchestra.run.smithi113.stdout: "name": "cephfs" 2024-01-15T11:27:01.093 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:27:01.093 INFO:teuthology.orchestra.run.smithi113.stdout:] 2024-01-15T11:27:01.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.095+0000 7fb1c17af700 1 -- 172.21.15.113:0/398427316 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb1a806b500 msgr2=0x7fb1a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.095+0000 7fb1c17af700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb1a806b500 0x7fb1a806d9c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009130 tx=0x7fb1b0008030 comp rx=0 tx=0).stop 2024-01-15T11:27:01.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.095+0000 7fb1c17af700 1 -- 172.21.15.113:0/398427316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1bc0d0db0 msgr2=0x7fb1bc152470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.095+0000 7fb1c17af700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1bc0d0db0 0x7fb1bc152470 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac00c7f0 tx=0x7fb1ac00cbb0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.095+0000 7fb1c17af700 1 -- 172.21.15.113:0/398427316 shutdown_connections 2024-01-15T11:27:01.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.095+0000 7fb1c17af700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fb1a806b500 0x7fb1a806d9c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.095+0000 7fb1c17af700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb1bc15e1e0 0x7fb1bc155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.095+0000 7fb1c17af700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1bc0d0db0 0x7fb1bc152470 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.095+0000 7fb1c17af700 1 --2- 172.21.15.113:0/398427316 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fb1bc0d3a00 0x7fb1bc1555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.095+0000 7fb1c17af700 1 -- 172.21.15.113:0/398427316 >> 172.21.15.113:0/398427316 conn(0x7fb1bc0b5b50 msgr2=0x7fb1bc0b6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:01.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.095+0000 7fb1c17af700 1 -- 172.21.15.113:0/398427316 shutdown_connections 2024-01-15T11:27:01.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.095+0000 7fb1c17af700 1 -- 172.21.15.113:0/398427316 wait complete. 2024-01-15T11:27:01.107 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T11:27:01.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.232+0000 7f250d204700 1 Processor -- start 2024-01-15T11:27:01.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.232+0000 7f250d204700 1 -- start start 2024-01-15T11:27:01.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.232+0000 7f250d204700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f250815e0c0 0x7f25080afb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.232+0000 7f250d204700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f25080746a0 0x7f25080add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.232+0000 7f250d204700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250815e4a0 con 0x7f25080b3bf0 2024-01-15T11:27:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.232+0000 7f250d204700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250806aff0 con 0x7f25080746a0 2024-01-15T11:27:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.232+0000 7f250d204700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250806b2d0 con 0x7f250815e0c0 2024-01-15T11:27:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.233+0000 7f250759e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f25080746a0 0x7f25080add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.233+0000 7f250759e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f25080746a0 0x7f25080add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34978/0 (socket says 172.21.15.113:34978) 2024-01-15T11:27:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.233+0000 7f250759e700 1 -- 172.21.15.113:0/447419176 learned_addr learned my addr 172.21.15.113:0/447419176 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.233+0000 7f2506d9d700 1 --2- 172.21.15.113:0/447419176 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f250815e0c0 0x7f25080afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.233+0000 7f250759e700 1 -- 172.21.15.113:0/447419176 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f250815e0c0 msgr2=0x7f25080afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.233+0000 7f250759e700 1 --2- 172.21.15.113:0/447419176 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f250815e0c0 0x7f25080afb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.233+0000 7f250759e700 1 -- 172.21.15.113:0/447419176 >> v1:172.21.15.141:6789/0 conn(0x7f25080b3bf0 legacy=0x7f2508073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.233+0000 7f250759e700 1 -- 172.21.15.113:0/447419176 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250806b5b0 con 0x7f25080746a0 2024-01-15T11:27:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.233+0000 7f250759e700 1 --2- 172.21.15.113:0/447419176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25080746a0 0x7f25080add40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2500009e50 tx=0x7f250000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9fd713c7438838d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.233+0000 7f2505d9b700 1 -- 172.21.15.113:0/447419176 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250000e070 con 0x7f25080746a0 2024-01-15T11:27:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.233+0000 7f2505d9b700 1 -- 172.21.15.113:0/447419176 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250000b9c0 con 0x7f25080746a0 2024-01-15T11:27:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.233+0000 7f2505d9b700 1 -- 172.21.15.113:0/447419176 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250000bd70 con 0x7f25080746a0 2024-01-15T11:27:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.234+0000 7f250d204700 1 -- 172.21.15.113:0/447419176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25080746a0 msgr2=0x7f25080add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.234+0000 7f250d204700 1 --2- 172.21.15.113:0/447419176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25080746a0 0x7f25080add40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2500009e50 tx=0x7f250000b330 comp rx=0 tx=0).stop 2024-01-15T11:27:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.234+0000 7f250d204700 1 -- 172.21.15.113:0/447419176 shutdown_connections 2024-01-15T11:27:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.234+0000 7f250d204700 1 --2- 172.21.15.113:0/447419176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25080746a0 0x7f25080add40 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.234+0000 7f250d204700 1 --2- 172.21.15.113:0/447419176 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f250815e0c0 0x7f25080afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.234+0000 7f250d204700 1 -- 172.21.15.113:0/447419176 >> 172.21.15.113:0/447419176 conn(0x7f25080d0c00 msgr2=0x7f25080d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.234+0000 7f250d204700 1 -- 172.21.15.113:0/447419176 shutdown_connections 2024-01-15T11:27:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.234+0000 7f250d204700 1 -- 172.21.15.113:0/447419176 wait complete. 2024-01-15T11:27:01.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.235+0000 7f250d204700 1 Processor -- start 2024-01-15T11:27:01.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.236+0000 7f250d204700 1 -- start start 2024-01-15T11:27:01.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.236+0000 7f250d204700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25080746a0 0x7f2508144750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.236+0000 7f250d204700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25080b3bf0 0x7f25080cfb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.236+0000 7f250d204700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f250815e0c0 0x7f250815c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.236+0000 7f250d204700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25080ceb70 con 0x7f250815e0c0 2024-01-15T11:27:01.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.236+0000 7f250d204700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2508152ac0 con 0x7f25080746a0 2024-01-15T11:27:01.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.236+0000 7f250d204700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25080cdf60 con 0x7f25080b3bf0 2024-01-15T11:27:01.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250759e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f250815e0c0 0x7f250815c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250759e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f250815e0c0 0x7f250815c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34994/0 (socket says 172.21.15.113:34994) 2024-01-15T11:27:01.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250759e700 1 -- 172.21.15.113:0/1846169508 learned_addr learned my addr 172.21.15.113:0/1846169508 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:01.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250659c700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25080b3bf0 0x7f25080cfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f2506d9d700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25080746a0 0x7f2508144750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250759e700 1 -- 172.21.15.113:0/1846169508 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25080b3bf0 msgr2=0x7f25080cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250759e700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25080b3bf0 0x7f25080cfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250759e700 1 -- 172.21.15.113:0/1846169508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25080746a0 msgr2=0x7f2508144750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250759e700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25080746a0 0x7f2508144750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250759e700 1 -- 172.21.15.113:0/1846169508 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2500009b30 con 0x7f250815e0c0 2024-01-15T11:27:01.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250659c700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25080b3bf0 0x7f25080cfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:27:01.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250759e700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f250815e0c0 0x7f250815c180 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2500012000 tx=0x7f2500004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:01.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f24ef7fe700 1 -- 172.21.15.113:0/1846169508 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000099b0 con 0x7f250815e0c0 2024-01-15T11:27:01.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f24ef7fe700 1 -- 172.21.15.113:0/1846169508 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25000092e0 con 0x7f250815e0c0 2024-01-15T11:27:01.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f24ef7fe700 1 -- 172.21.15.113:0/1846169508 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250001c640 con 0x7f250815e0c0 2024-01-15T11:27:01.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250d204700 1 -- 172.21.15.113:0/1846169508 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25080ce240 con 0x7f250815e0c0 2024-01-15T11:27:01.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.237+0000 7f250d204700 1 -- 172.21.15.113:0/1846169508 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25080ce800 con 0x7f250815e0c0 2024-01-15T11:27:01.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.238+0000 7f250d204700 1 -- 172.21.15.113:0/1846169508 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2508152c30 con 0x7f250815e0c0 2024-01-15T11:27:01.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.239+0000 7f24ef7fe700 1 -- 172.21.15.113:0/1846169508 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f250001c7a0 con 0x7f250815e0c0 2024-01-15T11:27:01.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.239+0000 7f24ef7fe700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f24f406b530 0x7f24f406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.239+0000 7f24ef7fe700 1 -- 172.21.15.113:0/1846169508 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f250008b8a0 con 0x7f250815e0c0 2024-01-15T11:27:01.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.239+0000 7f2506d9d700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f24f406b530 0x7f24f406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.239+0000 7f2506d9d700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f24f406b530 0x7f24f406d9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009930 tx=0x7f24f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:01.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.244+0000 7f24ef7fe700 1 -- 172.21.15.113:0/1846169508 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2508152c30 con 0x7f250815e0c0 2024-01-15T11:27:01.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.465+0000 7f250d204700 1 -- 172.21.15.113:0/1846169508 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f25080b4050 con 0x7f24f406b530 2024-01-15T11:27:01.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.468+0000 7f24ef7fe700 1 -- 172.21.15.113:0/1846169508 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f24f8009e00 con 0x7f24f406b530 2024-01-15T11:27:01.468 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:27:01.468 INFO:teuthology.orchestra.run.smithi113.stdout:[ 2024-01-15T11:27:01.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:27:01.468 INFO:teuthology.orchestra.run.smithi113.stdout: "name": "sv1" 2024-01-15T11:27:01.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-15T11:27:01.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:27:01.468 INFO:teuthology.orchestra.run.smithi113.stdout: "name": "ss1clone1" 2024-01-15T11:27:01.469 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:27:01.469 INFO:teuthology.orchestra.run.smithi113.stdout:] 2024-01-15T11:27:01.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.470+0000 7f250d204700 1 -- 172.21.15.113:0/1846169508 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f24f406b530 msgr2=0x7f24f406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.470+0000 7f250d204700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f24f406b530 0x7f24f406d9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009930 tx=0x7f24f8008040 comp rx=0 tx=0).stop 2024-01-15T11:27:01.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.470+0000 7f250d204700 1 -- 172.21.15.113:0/1846169508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f250815e0c0 msgr2=0x7f250815c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.470+0000 7f250d204700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f250815e0c0 0x7f250815c180 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2500012000 tx=0x7f2500004ad0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.470+0000 7f250d204700 1 -- 172.21.15.113:0/1846169508 shutdown_connections 2024-01-15T11:27:01.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.470+0000 7f250d204700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f24f406b530 0x7f24f406d9f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.470+0000 7f250d204700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f250815e0c0 0x7f250815c180 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.470+0000 7f250d204700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f25080746a0 0x7f2508144750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.470+0000 7f250d204700 1 --2- 172.21.15.113:0/1846169508 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f25080b3bf0 0x7f25080cfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.470+0000 7f250d204700 1 -- 172.21.15.113:0/1846169508 >> 172.21.15.113:0/1846169508 conn(0x7f25080d0c00 msgr2=0x7f25080b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:01.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.470+0000 7f250d204700 1 -- 172.21.15.113:0/1846169508 shutdown_connections 2024-01-15T11:27:01.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.470+0000 7f250d204700 1 -- 172.21.15.113:0/1846169508 wait complete. 2024-01-15T11:27:01.482 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T11:27:01.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.609+0000 7fcc1ea88700 1 Processor -- start 2024-01-15T11:27:01.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.609+0000 7fcc1ea88700 1 -- start start 2024-01-15T11:27:01.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.610+0000 7fcc1ea88700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc1815e020 0x7fcc180b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.610+0000 7fcc1ea88700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fcc180cfae0 0x7fcc180af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.610+0000 7fcc1ea88700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc180b1ba0 con 0x7fcc180b1720 2024-01-15T11:27:01.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.610+0000 7fcc1ea88700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc180cff70 con 0x7fcc180cfae0 2024-01-15T11:27:01.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.610+0000 7fcc1ea88700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc18161c80 con 0x7fcc1815e020 2024-01-15T11:27:01.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.610+0000 7fcc17fff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fcc180b1720 0x7fcc180cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51930/0 (socket says 172.21.15.113:51930) 2024-01-15T11:27:01.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.610+0000 7fcc17fff700 1 -- 172.21.15.113:0/3592895947 learned_addr learned my addr 172.21.15.113:0/3592895947 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:27:01.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.610+0000 7fcc1d025700 1 --2- 172.21.15.113:0/3592895947 >> v2:172.21.15.141:3300/0 conn(0x7fcc180cfae0 0x7fcc180af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.610+0000 7fcc1c824700 1 --2- 172.21.15.113:0/3592895947 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc1815e020 0x7fcc180b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.610+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579876041 0 0) 0x7fcc180b1ba0 con 0x7fcc180b1720 2024-01-15T11:27:01.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.610+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc080035b0 con 0x7fcc180b1720 2024-01-15T11:27:01.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 612448415 0 0) 0x7fcc080035b0 con 0x7fcc180b1720 2024-01-15T11:27:01.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc1815e020 msgr2=0x7fcc180b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:01.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 --2- 172.21.15.113:0/3592895947 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc1815e020 0x7fcc180b11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 >> v2:172.21.15.141:3300/0 conn(0x7fcc180cfae0 msgr2=0x7fcc180af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:01.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 --2- 172.21.15.113:0/3592895947 >> v2:172.21.15.141:3300/0 conn(0x7fcc180cfae0 0x7fcc180af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc18161f40 con 0x7fcc180b1720 2024-01-15T11:27:01.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fcc04001670 con 0x7fcc180b1720 2024-01-15T11:27:01.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 >> v1:172.21.15.113:6789/0 conn(0x7fcc180b1720 legacy=0x7fcc180cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 --2- 172.21.15.113:0/3592895947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc08003a00 0x7fcc08003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 --2- 172.21.15.113:0/3592895947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc08004850 0x7fcc08004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 --2- 172.21.15.113:0/3592895947 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc1815e020 0x7fcc08005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180cff70 con 0x7fcc08003a00 2024-01-15T11:27:01.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc18161c80 con 0x7fcc08004850 2024-01-15T11:27:01.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0800ba10 con 0x7fcc1815e020 2024-01-15T11:27:01.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc1c824700 1 --2- 172.21.15.113:0/3592895947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc08003a00 0x7fcc08003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc1d025700 1 --2- 172.21.15.113:0/3592895947 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc1815e020 0x7fcc08005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc17fff700 1 --2- 172.21.15.113:0/3592895947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc08004850 0x7fcc08004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc1c824700 1 -- 172.21.15.113:0/3592895947 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc1815e020 msgr2=0x7fcc08005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:01.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc1c824700 1 --2- 172.21.15.113:0/3592895947 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc1815e020 0x7fcc08005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc1c824700 1 -- 172.21.15.113:0/3592895947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc08004850 msgr2=0x7fcc08004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:01.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc1c824700 1 --2- 172.21.15.113:0/3592895947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc08004850 0x7fcc08004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc1c824700 1 -- 172.21.15.113:0/3592895947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc080035b0 con 0x7fcc08003a00 2024-01-15T11:27:01.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc1c824700 1 --2- 172.21.15.113:0/3592895947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc08003a00 0x7fcc08003e20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009b90 tx=0x7fcc08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f5a426ae017bba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:01.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00e6c0 con 0x7fcc08003a00 2024-01-15T11:27:01.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0c00e820 con 0x7fcc08003a00 2024-01-15T11:27:01.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.611+0000 7fcc177fe700 1 -- 172.21.15.113:0/3592895947 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00eb90 con 0x7fcc08003a00 2024-01-15T11:27:01.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.612+0000 7fcc1ea88700 1 -- 172.21.15.113:0/3592895947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc08003a00 msgr2=0x7fcc08003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.612+0000 7fcc1ea88700 1 --2- 172.21.15.113:0/3592895947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc08003a00 0x7fcc08003e20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009b90 tx=0x7fcc08005880 comp rx=0 tx=0).stop 2024-01-15T11:27:01.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.612+0000 7fcc1d025700 1 -- 172.21.15.113:0/3592895947 reap_dead start 2024-01-15T11:27:01.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.612+0000 7fcc1ea88700 1 -- 172.21.15.113:0/3592895947 shutdown_connections 2024-01-15T11:27:01.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.612+0000 7fcc1ea88700 1 -- 172.21.15.113:0/3592895947 >> 172.21.15.113:0/3592895947 conn(0x7fcc18072710 msgr2=0x7fcc18072b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:01.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.612+0000 7fcc1ea88700 1 -- 172.21.15.113:0/3592895947 shutdown_connections 2024-01-15T11:27:01.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.612+0000 7fcc1ea88700 1 -- 172.21.15.113:0/3592895947 wait complete. 2024-01-15T11:27:01.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.613+0000 7fcc1ea88700 1 Processor -- start 2024-01-15T11:27:01.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.613+0000 7fcc1ea88700 1 -- start start 2024-01-15T11:27:01.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1ea88700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc08003a00 0x7fcc18099950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1ea88700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc180b1e70 0x7fcc180b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1ea88700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc180b56c0 0x7fcc18158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1ea88700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180b5b70 con 0x7fcc08003a00 2024-01-15T11:27:01.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1ea88700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc18099e90 con 0x7fcc180b56c0 2024-01-15T11:27:01.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1ea88700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc18157270 con 0x7fcc180b1e70 2024-01-15T11:27:01.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1c824700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc08003a00 0x7fcc18099950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1c824700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc08003a00 0x7fcc18099950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35018/0 (socket says 172.21.15.113:35018) 2024-01-15T11:27:01.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1c824700 1 -- 172.21.15.113:0/283746907 learned_addr learned my addr 172.21.15.113:0/283746907 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:01.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1d025700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc180b56c0 0x7fcc18158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc17fff700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc180b1e70 0x7fcc180b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1c824700 1 -- 172.21.15.113:0/283746907 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc180b1e70 msgr2=0x7fcc180b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1c824700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc180b1e70 0x7fcc180b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1c824700 1 -- 172.21.15.113:0/283746907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc180b56c0 msgr2=0x7fcc18158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1c824700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc180b56c0 0x7fcc18158030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.614+0000 7fcc1c824700 1 -- 172.21.15.113:0/283746907 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc18157550 con 0x7fcc08003a00 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.615+0000 7fcc17fff700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc180b1e70 0x7fcc180b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.615+0000 7fcc1c824700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc08003a00 0x7fcc18099950 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00e4b0 tx=0x7fcc0c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.615+0000 7fcc157fa700 1 -- 172.21.15.113:0/283746907 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c021070 con 0x7fcc08003a00 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.615+0000 7fcc1ea88700 1 -- 172.21.15.113:0/283746907 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc18157830 con 0x7fcc08003a00 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.615+0000 7fcc157fa700 1 -- 172.21.15.113:0/283746907 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0c004540 con 0x7fcc08003a00 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.615+0000 7fcc157fa700 1 -- 172.21.15.113:0/283746907 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00e6c0 con 0x7fcc08003a00 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.615+0000 7fcc1ea88700 1 -- 172.21.15.113:0/283746907 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc180d09c0 con 0x7fcc08003a00 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.616+0000 7fcc1ea88700 1 -- 172.21.15.113:0/283746907 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc18157e80 con 0x7fcc08003a00 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.616+0000 7fcc157fa700 1 -- 172.21.15.113:0/283746907 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fcc0c025070 con 0x7fcc08003a00 2024-01-15T11:27:01.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.616+0000 7fcc157fa700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fcbfc06b500 0x7fcbfc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:01.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.617+0000 7fcc157fa700 1 -- 172.21.15.113:0/283746907 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc0c08b080 con 0x7fcc08003a00 2024-01-15T11:27:01.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.617+0000 7fcc17fff700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fcbfc06b500 0x7fcbfc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:01.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.617+0000 7fcc17fff700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fcbfc06b500 0x7fcbfc06d9c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcc18156130 tx=0x7fcc04000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:01.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.621+0000 7fcc157fa700 1 -- 172.21.15.113:0/283746907 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc0c055c80 con 0x7fcc08003a00 2024-01-15T11:27:01.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.847+0000 7fcc1ea88700 1 -- 172.21.15.113:0/283746907 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcc181573e0 con 0x7fcbfc06b500 2024-01-15T11:27:01.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.858+0000 7fcc157fa700 1 -- 172.21.15.113:0/283746907 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fcc181573e0 con 0x7fcbfc06b500 2024-01-15T11:27:01.857 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:27:01.858 INFO:teuthology.orchestra.run.smithi113.stdout:[ 2024-01-15T11:27:01.858 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-15T11:27:01.858 INFO:teuthology.orchestra.run.smithi113.stdout: "name": "ss1" 2024-01-15T11:27:01.858 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-15T11:27:01.858 INFO:teuthology.orchestra.run.smithi113.stdout:] 2024-01-15T11:27:01.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.860+0000 7fcc1ea88700 1 -- 172.21.15.113:0/283746907 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fcbfc06b500 msgr2=0x7fcbfc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.860+0000 7fcc1ea88700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fcbfc06b500 0x7fcbfc06d9c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcc18156130 tx=0x7fcc04000ed0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.860+0000 7fcc1ea88700 1 -- 172.21.15.113:0/283746907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc08003a00 msgr2=0x7fcc18099950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:01.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.860+0000 7fcc1ea88700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc08003a00 0x7fcc18099950 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00e4b0 tx=0x7fcc0c0047c0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.860+0000 7fcc1ea88700 1 -- 172.21.15.113:0/283746907 shutdown_connections 2024-01-15T11:27:01.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.860+0000 7fcc1ea88700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fcbfc06b500 0x7fcbfc06d9c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.860+0000 7fcc1ea88700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc180b56c0 0x7fcc18158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.860+0000 7fcc1ea88700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc08003a00 0x7fcc18099950 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.860+0000 7fcc1ea88700 1 --2- 172.21.15.113:0/283746907 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fcc180b1e70 0x7fcc180b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:01.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.860+0000 7fcc1ea88700 1 -- 172.21.15.113:0/283746907 >> 172.21.15.113:0/283746907 conn(0x7fcc18072710 msgr2=0x7fcc180c42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:01.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.860+0000 7fcc1ea88700 1 -- 172.21.15.113:0/283746907 shutdown_connections 2024-01-15T11:27:01.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:01.860+0000 7fcc1ea88700 1 -- 172.21.15.113:0/283746907 wait complete. 2024-01-15T11:27:01.872 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T11:27:01.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.000+0000 7fe0e6622700 1 Processor -- start 2024-01-15T11:27:01.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.000+0000 7fe0e6622700 1 -- start start 2024-01-15T11:27:02.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.000+0000 7fe0e6622700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fe0e00b4660 0x7fe0e00b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0e6622700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e006c4c0 con 0x7fe0e00cf860 2024-01-15T11:27:02.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0e6622700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e006c7a0 con 0x7fe0e01447e0 2024-01-15T11:27:02.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0e6622700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e0144bc0 con 0x7fe0e00b4660 2024-01-15T11:27:02.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0e4bbf700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fe0e00cf860 0x7fe0e00cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51932/0 (socket says 172.21.15.113:51932) 2024-01-15T11:27:02.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0e4bbf700 1 -- 172.21.15.113:0/4230639888 learned_addr learned my addr 172.21.15.113:0/4230639888 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:27:02.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0df7fe700 1 --2- 172.21.15.113:0/4230639888 >> v2:172.21.15.141:3300/0 conn(0x7fe0e00b4660 0x7fe0e00b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258029667 0 0) 0x7fe0e006c4c0 con 0x7fe0e00cf860 2024-01-15T11:27:02.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c80035b0 con 0x7fe0e00cf860 2024-01-15T11:27:02.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046996130 0 0) 0x7fe0e006c7a0 con 0x7fe0e01447e0 2024-01-15T11:27:02.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 --> v1:172.21.15.141:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e006c4c0 con 0x7fe0e01447e0 2024-01-15T11:27:02.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 306883057 0 0) 0x7fe0c80035b0 con 0x7fe0e00cf860 2024-01-15T11:27:02.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 >> v2:172.21.15.141:3300/0 conn(0x7fe0e00b4660 msgr2=0x7fe0e00b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:02.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 --2- 172.21.15.113:0/4230639888 >> v2:172.21.15.141:3300/0 conn(0x7fe0e00b4660 0x7fe0e00b4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 >> v1:172.21.15.141:6789/0 conn(0x7fe0e01447e0 legacy=0x7fe0e006ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e00cf060 con 0x7fe0e00cf860 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fe0d80019f0 con 0x7fe0e00cf860 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 >> v1:172.21.15.113:6789/0 conn(0x7fe0e00cf860 legacy=0x7fe0e00cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 --2- 172.21.15.113:0/4230639888 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0c8003be0 0x7fe0c8003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 --2- 172.21.15.113:0/4230639888 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe0c8004850 0x7fe0c8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 --2- 172.21.15.113:0/4230639888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c8005210 0x7fe0c800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e0144bc0 con 0x7fe0c8005210 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c800c0e0 con 0x7fe0c8003be0 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c800c2a0 con 0x7fe0c8004850 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0e4bbf700 1 --2- 172.21.15.113:0/4230639888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c8005210 0x7fe0c800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.001+0000 7fe0dffff700 1 --2- 172.21.15.113:0/4230639888 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0c8003be0 0x7fe0c8003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0df7fe700 1 --2- 172.21.15.113:0/4230639888 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe0c8004850 0x7fe0c8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0e4bbf700 1 -- 172.21.15.113:0/4230639888 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe0c8004850 msgr2=0x7fe0c8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:02.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0e4bbf700 1 --2- 172.21.15.113:0/4230639888 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe0c8004850 0x7fe0c8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0e4bbf700 1 -- 172.21.15.113:0/4230639888 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0c8003be0 msgr2=0x7fe0c8003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0e4bbf700 1 --2- 172.21.15.113:0/4230639888 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0c8003be0 0x7fe0c8003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0e4bbf700 1 -- 172.21.15.113:0/4230639888 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80035b0 con 0x7fe0c8005210 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0e4bbf700 1 -- 172.21.15.113:0/4230639888 reap_dead start 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0e4bbf700 1 --2- 172.21.15.113:0/4230639888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c8005210 0x7fe0c800ba10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800a710 tx=0x7fe0c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5af4ee07a1fd27 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d80072d0 con 0x7fe0c8005210 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d8007430 con 0x7fe0c8005210 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0deffd700 1 -- 172.21.15.113:0/4230639888 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d80077c0 con 0x7fe0c8005210 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0e6622700 1 -- 172.21.15.113:0/4230639888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c8005210 msgr2=0x7fe0c800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0e6622700 1 --2- 172.21.15.113:0/4230639888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c8005210 0x7fe0c800ba10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800a710 tx=0x7fe0c8005880 comp rx=0 tx=0).stop 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0e6622700 1 -- 172.21.15.113:0/4230639888 shutdown_connections 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0e6622700 1 --2- 172.21.15.113:0/4230639888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c8005210 0x7fe0c800ba10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.002+0000 7fe0e6622700 1 -- 172.21.15.113:0/4230639888 >> 172.21.15.113:0/4230639888 conn(0x7fe0e00cd620 msgr2=0x7fe0e00cda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:02.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.003+0000 7fe0e6622700 1 -- 172.21.15.113:0/4230639888 shutdown_connections 2024-01-15T11:27:02.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.003+0000 7fe0e6622700 1 -- 172.21.15.113:0/4230639888 wait complete. 2024-01-15T11:27:02.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.004+0000 7fe0e6622700 1 Processor -- start 2024-01-15T11:27:02.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.004+0000 7fe0e6622700 1 -- start start 2024-01-15T11:27:02.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e6622700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0c8005210 0x7fe0e0153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e6622700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe0e0142b70 0x7fe0e01f6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e6622700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0e013f6e0 0x7fe0e013fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e6622700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e01400a0 con 0x7fe0e013f6e0 2024-01-15T11:27:02.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e6622700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e0153f60 con 0x7fe0c8005210 2024-01-15T11:27:02.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e6622700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e0154240 con 0x7fe0e0142b70 2024-01-15T11:27:02.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e4bbf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0e013f6e0 0x7fe0e013fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e4bbf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0e013f6e0 0x7fe0e013fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35024/0 (socket says 172.21.15.113:35024) 2024-01-15T11:27:02.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e4bbf700 1 -- 172.21.15.113:0/4022314687 learned_addr learned my addr 172.21.15.113:0/4022314687 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:02.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0dffff700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0c8005210 0x7fe0e0153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0df7fe700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe0e0142b70 0x7fe0e01f6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e4bbf700 1 -- 172.21.15.113:0/4022314687 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe0e0142b70 msgr2=0x7fe0e01f6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e4bbf700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe0e0142b70 0x7fe0e01f6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e4bbf700 1 -- 172.21.15.113:0/4022314687 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0c8005210 msgr2=0x7fe0e0153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e4bbf700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0c8005210 0x7fe0e0153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e4bbf700 1 -- 172.21.15.113:0/4022314687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e0154520 con 0x7fe0e013f6e0 2024-01-15T11:27:02.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e4bbf700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0e013f6e0 0x7fe0e013fb60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00b5ee0 tx=0x7fe0d8001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:02.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0dcff9700 1 -- 172.21.15.113:0/4022314687 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d80079b0 con 0x7fe0e013f6e0 2024-01-15T11:27:02.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0dcff9700 1 -- 172.21.15.113:0/4022314687 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d8007b10 con 0x7fe0e013f6e0 2024-01-15T11:27:02.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e6622700 1 -- 172.21.15.113:0/4022314687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e0154800 con 0x7fe0e013f6e0 2024-01-15T11:27:02.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0dcff9700 1 -- 172.21.15.113:0/4022314687 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d801f670 con 0x7fe0e013f6e0 2024-01-15T11:27:02.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.005+0000 7fe0e6622700 1 -- 172.21.15.113:0/4022314687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e0142ff0 con 0x7fe0e013f6e0 2024-01-15T11:27:02.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.006+0000 7fe0e6622700 1 -- 172.21.15.113:0/4022314687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e0062a30 con 0x7fe0e013f6e0 2024-01-15T11:27:02.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.007+0000 7fe0dcff9700 1 -- 172.21.15.113:0/4022314687 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fe0d801f7d0 con 0x7fe0e013f6e0 2024-01-15T11:27:02.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.007+0000 7fe0dcff9700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe0cc06b500 0x7fe0cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.007+0000 7fe0dcff9700 1 -- 172.21.15.113:0/4022314687 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0d808b180 con 0x7fe0e013f6e0 2024-01-15T11:27:02.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.007+0000 7fe0dffff700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe0cc06b500 0x7fe0cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.008+0000 7fe0dffff700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe0cc06b500 0x7fe0cc06d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0000c00 tx=0x7fe0d0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:02.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.012+0000 7fe0dcff9700 1 -- 172.21.15.113:0/4022314687 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0d8055eb0 con 0x7fe0e013f6e0 2024-01-15T11:27:02.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.244+0000 7fe0e6622700 1 -- 172.21.15.113:0/4022314687 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe0e006dcf0 con 0x7fe0cc06b500 2024-01-15T11:27:02.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.267+0000 7fe0dcff9700 1 -- 172.21.15.113:0/4022314687 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe0d00016e0 con 0x7fe0cc06b500 2024-01-15T11:27:02.267 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:27:02.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.269+0000 7fe0e6622700 1 -- 172.21.15.113:0/4022314687 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe0cc06b500 msgr2=0x7fe0cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.269+0000 7fe0e6622700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe0cc06b500 0x7fe0cc06d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0000c00 tx=0x7fe0d0009040 comp rx=0 tx=0).stop 2024-01-15T11:27:02.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.269+0000 7fe0e6622700 1 -- 172.21.15.113:0/4022314687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0e013f6e0 msgr2=0x7fe0e013fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.269+0000 7fe0e6622700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0e013f6e0 0x7fe0e013fb60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00b5ee0 tx=0x7fe0d8001cb0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.269+0000 7fe0e6622700 1 -- 172.21.15.113:0/4022314687 shutdown_connections 2024-01-15T11:27:02.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.269+0000 7fe0e6622700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fe0cc06b500 0x7fe0cc06d9c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.269+0000 7fe0e6622700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0e013f6e0 0x7fe0e013fb60 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.269+0000 7fe0e6622700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe0c8005210 0x7fe0e0153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.269+0000 7fe0e6622700 1 --2- 172.21.15.113:0/4022314687 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fe0e0142b70 0x7fe0e01f6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.269+0000 7fe0e6622700 1 -- 172.21.15.113:0/4022314687 >> 172.21.15.113:0/4022314687 conn(0x7fe0e00cd620 msgr2=0x7fe0e0071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:02.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.270+0000 7fe0e6622700 1 -- 172.21.15.113:0/4022314687 shutdown_connections 2024-01-15T11:27:02.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.270+0000 7fe0e6622700 1 -- 172.21.15.113:0/4022314687 wait complete. 2024-01-15T11:27:02.281 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T11:27:02.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.406+0000 7fd60154e700 1 Processor -- start 2024-01-15T11:27:02.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.407+0000 7fd60154e700 1 -- start start 2024-01-15T11:27:02.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.407+0000 7fd60154e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd5fc151620 0x7fd5fc0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.407+0000 7fd60154e700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd5fc15df60 0x7fd5fc15e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.407+0000 7fd60154e700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5fc0b3f60 con 0x7fd5fc0b3470 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.407+0000 7fd60154e700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5fc15e8f0 con 0x7fd5fc151620 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.407+0000 7fd60154e700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5fc06d190 con 0x7fd5fc15df60 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.407+0000 7fd5faffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd5fc151620 0x7fd5fc0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.408+0000 7fd5faffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd5fc151620 0x7fd5fc0cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35026/0 (socket says 172.21.15.113:35026) 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.408+0000 7fd5faffd700 1 -- 172.21.15.113:0/3437444826 learned_addr learned my addr 172.21.15.113:0/3437444826 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.408+0000 7fd5fb7fe700 1 --2- 172.21.15.113:0/3437444826 >> v2:172.21.15.141:3300/0 conn(0x7fd5fc15df60 0x7fd5fc15e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.408+0000 7fd5faffd700 1 -- 172.21.15.113:0/3437444826 >> v2:172.21.15.141:3300/0 conn(0x7fd5fc15df60 msgr2=0x7fd5fc15e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.408+0000 7fd5faffd700 1 --2- 172.21.15.113:0/3437444826 >> v2:172.21.15.141:3300/0 conn(0x7fd5fc15df60 0x7fd5fc15e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.408+0000 7fd5faffd700 1 -- 172.21.15.113:0/3437444826 >> v1:172.21.15.141:6789/0 conn(0x7fd5fc0b3470 legacy=0x7fd5fc070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.408+0000 7fd5faffd700 1 -- 172.21.15.113:0/3437444826 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc06d750 con 0x7fd5fc151620 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.408+0000 7fd5faffd700 1 --2- 172.21.15.113:0/3437444826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5fc151620 0x7fd5fc0cf230 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc15b9a0 tx=0x7fd5f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2dea587a313faed server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.408+0000 7fd5f9ffb700 1 -- 172.21.15.113:0/3437444826 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f0004230 con 0x7fd5fc151620 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.408+0000 7fd5f9ffb700 1 -- 172.21.15.113:0/3437444826 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f0004390 con 0x7fd5fc151620 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.408+0000 7fd5f9ffb700 1 -- 172.21.15.113:0/3437444826 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f00036a0 con 0x7fd5fc151620 2024-01-15T11:27:02.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.409+0000 7fd60154e700 1 -- 172.21.15.113:0/3437444826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5fc151620 msgr2=0x7fd5fc0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.409+0000 7fd60154e700 1 --2- 172.21.15.113:0/3437444826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5fc151620 0x7fd5fc0cf230 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc15b9a0 tx=0x7fd5f0009bd0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.409+0000 7fd60154e700 1 -- 172.21.15.113:0/3437444826 shutdown_connections 2024-01-15T11:27:02.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.409+0000 7fd60154e700 1 --2- 172.21.15.113:0/3437444826 >> v2:172.21.15.141:3300/0 conn(0x7fd5fc15df60 0x7fd5fc15e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.409+0000 7fd60154e700 1 --2- 172.21.15.113:0/3437444826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5fc151620 0x7fd5fc0cf230 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.409+0000 7fd60154e700 1 -- 172.21.15.113:0/3437444826 >> 172.21.15.113:0/3437444826 conn(0x7fd5fc06b410 msgr2=0x7fd5fc06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:02.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.409+0000 7fd60154e700 1 -- 172.21.15.113:0/3437444826 shutdown_connections 2024-01-15T11:27:02.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.409+0000 7fd60154e700 1 -- 172.21.15.113:0/3437444826 wait complete. 2024-01-15T11:27:02.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.410+0000 7fd60154e700 1 Processor -- start 2024-01-15T11:27:02.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.410+0000 7fd60154e700 1 -- start start 2024-01-15T11:27:02.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd60154e700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd5fc0b3470 0x7fd5fc14a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd60154e700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5fc151620 0x7fd5fc1455d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd60154e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5fc15df60 0x7fd5fc0cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd60154e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc148da0 con 0x7fd5fc15df60 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd60154e700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc14b7a0 con 0x7fd5fc151620 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd60154e700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc149630 con 0x7fd5fc0b3470 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5fb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5fc15df60 0x7fd5fc0cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5fb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5fc15df60 0x7fd5fc0cd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35038/0 (socket says 172.21.15.113:35038) 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5fb7fe700 1 -- 172.21.15.113:0/22685944 learned_addr learned my addr 172.21.15.113:0/22685944 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5fa7fc700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5fc151620 0x7fd5fc1455d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5faffd700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd5fc0b3470 0x7fd5fc14a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5fb7fe700 1 -- 172.21.15.113:0/22685944 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd5fc0b3470 msgr2=0x7fd5fc14a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5fb7fe700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd5fc0b3470 0x7fd5fc14a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5fb7fe700 1 -- 172.21.15.113:0/22685944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5fc151620 msgr2=0x7fd5fc1455d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5fb7fe700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5fc151620 0x7fd5fc1455d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5fb7fe700 1 -- 172.21.15.113:0/22685944 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f0009550 con 0x7fd5fc15df60 2024-01-15T11:27:02.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5fa7fc700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5fc151620 0x7fd5fc1455d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:27:02.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5faffd700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd5fc0b3470 0x7fd5fc14a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T11:27:02.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5fb7fe700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5fc15df60 0x7fd5fc0cd790 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400cc30 tx=0x7fd5f400a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:02.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.411+0000 7fd5e37fe700 1 -- 172.21.15.113:0/22685944 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f400ab70 con 0x7fd5fc15df60 2024-01-15T11:27:02.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.412+0000 7fd5e37fe700 1 -- 172.21.15.113:0/22685944 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f400acd0 con 0x7fd5fc15df60 2024-01-15T11:27:02.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.412+0000 7fd5e37fe700 1 -- 172.21.15.113:0/22685944 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4003c40 con 0x7fd5fc15df60 2024-01-15T11:27:02.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.412+0000 7fd60154e700 1 -- 172.21.15.113:0/22685944 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc149910 con 0x7fd5fc15df60 2024-01-15T11:27:02.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.412+0000 7fd60154e700 1 -- 172.21.15.113:0/22685944 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5fc149ed0 con 0x7fd5fc15df60 2024-01-15T11:27:02.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.413+0000 7fd60154e700 1 -- 172.21.15.113:0/22685944 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5fc14a0f0 con 0x7fd5fc15df60 2024-01-15T11:27:02.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.413+0000 7fd5e37fe700 1 -- 172.21.15.113:0/22685944 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fd5f400f3f0 con 0x7fd5fc15df60 2024-01-15T11:27:02.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.413+0000 7fd5e37fe700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd5e806b500 0x7fd5e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.414+0000 7fd5faffd700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd5e806b500 0x7fd5e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.414+0000 7fd5e37fe700 1 -- 172.21.15.113:0/22685944 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5f408a1c0 con 0x7fd5fc15df60 2024-01-15T11:27:02.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.414+0000 7fd5faffd700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd5e806b500 0x7fd5e806d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc149440 tx=0x7fd5f0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:02.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.418+0000 7fd5e37fe700 1 -- 172.21.15.113:0/22685944 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5f4054c20 con 0x7fd5fc15df60 2024-01-15T11:27:02.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.645+0000 7fd60154e700 1 -- 172.21.15.113:0/22685944 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5fc0b3c80 con 0x7fd5e806b500 2024-01-15T11:27:02.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.661+0000 7fd5e37fe700 1 -- 172.21.15.113:0/22685944 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd5f0020070 con 0x7fd5e806b500 2024-01-15T11:27:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.663+0000 7fd60154e700 1 -- 172.21.15.113:0/22685944 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd5e806b500 msgr2=0x7fd5e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.663+0000 7fd60154e700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd5e806b500 0x7fd5e806d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc149440 tx=0x7fd5f0008030 comp rx=0 tx=0).stop 2024-01-15T11:27:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.663+0000 7fd60154e700 1 -- 172.21.15.113:0/22685944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5fc15df60 msgr2=0x7fd5fc0cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.663+0000 7fd60154e700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5fc15df60 0x7fd5fc0cd790 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400cc30 tx=0x7fd5f400a1f0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.663+0000 7fd60154e700 1 -- 172.21.15.113:0/22685944 shutdown_connections 2024-01-15T11:27:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.663+0000 7fd60154e700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd5e806b500 0x7fd5e806d9c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.663+0000 7fd60154e700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5fc15df60 0x7fd5fc0cd790 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.663+0000 7fd60154e700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd5fc0b3470 0x7fd5fc14a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.663+0000 7fd60154e700 1 --2- 172.21.15.113:0/22685944 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd5fc151620 0x7fd5fc1455d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.663+0000 7fd60154e700 1 -- 172.21.15.113:0/22685944 >> 172.21.15.113:0/22685944 conn(0x7fd5fc06b410 msgr2=0x7fd5fc15a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.663+0000 7fd60154e700 1 -- 172.21.15.113:0/22685944 shutdown_connections 2024-01-15T11:27:02.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.663+0000 7fd60154e700 1 -- 172.21.15.113:0/22685944 wait complete. 2024-01-15T11:27:02.664 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:27:02.664+0000 7fa4cbcac700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:27:02.664 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:27:02.664+0000 7fa4cbcac700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:27:02.664 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:27:02.664+0000 7fa4cbcac700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:27:02.665 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:27:02.664+0000 7fa4cbcac700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:27:02.665 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:27:02.664+0000 7fa4cbcac700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:27:02.678 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-15T11:27:02.679 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:27:02.678+0000 7fa4cacaa700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:27:02.679 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:27:02.678+0000 7fa4cacaa700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:27:02.679 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:27:02.678+0000 7fa4cacaa700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:27:02.679 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:27:02.678+0000 7fa4cacaa700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:27:02.679 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-15T11:27:02.678+0000 7fa4cacaa700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T11:27:02.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.830+0000 7fedfe98c700 1 Processor -- start 2024-01-15T11:27:02.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.830+0000 7fedfe98c700 1 -- start start 2024-01-15T11:27:02.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedfe98c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fedf80d3a40 0x7fedf8071610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedfe98c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedf8071b50 0x7fedf80b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedfe98c700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf80d29e0 con 0x7fedf815e0c0 2024-01-15T11:27:02.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedfe98c700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf8071fe0 con 0x7fedf80d3a40 2024-01-15T11:27:02.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedfe98c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80d2c80 con 0x7fedf8071b50 2024-01-15T11:27:02.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedf77fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fedf80d3a40 0x7fedf8071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedf77fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fedf80d3a40 0x7fedf8071610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35082/0 (socket says 172.21.15.113:35082) 2024-01-15T11:27:02.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedf77fe700 1 -- 172.21.15.113:0/2796686036 learned_addr learned my addr 172.21.15.113:0/2796686036 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedfcf29700 1 --2- 172.21.15.113:0/2796686036 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedf8071b50 0x7fedf80b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedf77fe700 1 -- 172.21.15.113:0/2796686036 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedf8071b50 msgr2=0x7fedf80b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedf77fe700 1 --2- 172.21.15.113:0/2796686036 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedf8071b50 0x7fedf80b6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedf77fe700 1 -- 172.21.15.113:0/2796686036 >> v1:172.21.15.141:6789/0 conn(0x7fedf815e0c0 legacy=0x7fedf80cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedf77fe700 1 -- 172.21.15.113:0/2796686036 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80d2f60 con 0x7fedf80d3a40 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.831+0000 7fedf77fe700 1 --2- 172.21.15.113:0/2796686036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedf80d3a40 0x7fedf8071610 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fedec0098f0 tx=0x7fedec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fde76fa6d6a572 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.832+0000 7fedf6ffd700 1 -- 172.21.15.113:0/2796686036 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec00b8a0 con 0x7fedf80d3a40 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.832+0000 7fedf6ffd700 1 -- 172.21.15.113:0/2796686036 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec00ba00 con 0x7fedf80d3a40 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.832+0000 7fedf6ffd700 1 -- 172.21.15.113:0/2796686036 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec00bd90 con 0x7fedf80d3a40 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.832+0000 7fedfe98c700 1 -- 172.21.15.113:0/2796686036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedf80d3a40 msgr2=0x7fedf8071610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.832+0000 7fedfe98c700 1 --2- 172.21.15.113:0/2796686036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedf80d3a40 0x7fedf8071610 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fedec0098f0 tx=0x7fedec009c00 comp rx=0 tx=0).stop 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.832+0000 7fedfe98c700 1 -- 172.21.15.113:0/2796686036 shutdown_connections 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.832+0000 7fedfe98c700 1 --2- 172.21.15.113:0/2796686036 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedf8071b50 0x7fedf80b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.832+0000 7fedfe98c700 1 --2- 172.21.15.113:0/2796686036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedf80d3a40 0x7fedf8071610 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.832+0000 7fedfe98c700 1 -- 172.21.15.113:0/2796686036 >> 172.21.15.113:0/2796686036 conn(0x7fedf806b620 msgr2=0x7fedf80d3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:02.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.832+0000 7fedfe98c700 1 -- 172.21.15.113:0/2796686036 shutdown_connections 2024-01-15T11:27:02.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.832+0000 7fedfe98c700 1 -- 172.21.15.113:0/2796686036 wait complete. 2024-01-15T11:27:02.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.834+0000 7fedfe98c700 1 Processor -- start 2024-01-15T11:27:02.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.834+0000 7fedfe98c700 1 -- start start 2024-01-15T11:27:02.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.835+0000 7fedfe98c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedf8071b50 0x7fedf80cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.835+0000 7fedfe98c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedf80d3a40 0x7fedf81606c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.835+0000 7fedfe98c700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedf815e0c0 0x7fedf80cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.835+0000 7fedfe98c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80cb220 con 0x7fedf8071b50 2024-01-15T11:27:02.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.835+0000 7fedfe98c700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80caf20 con 0x7fedf80d3a40 2024-01-15T11:27:02.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.835+0000 7fedfe98c700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80cc900 con 0x7fedf815e0c0 2024-01-15T11:27:02.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedf8071b50 0x7fedf80cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedf8071b50 0x7fedf80cdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35092/0 (socket says 172.21.15.113:35092) 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf7fff700 1 -- 172.21.15.113:0/1571218845 learned_addr learned my addr 172.21.15.113:0/1571218845 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedfcf29700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedf815e0c0 0x7fedf80cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf77fe700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedf80d3a40 0x7fedf81606c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf7fff700 1 -- 172.21.15.113:0/1571218845 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedf815e0c0 msgr2=0x7fedf80cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf7fff700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedf815e0c0 0x7fedf80cb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf7fff700 1 -- 172.21.15.113:0/1571218845 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedf80d3a40 msgr2=0x7fedf81606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf7fff700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedf80d3a40 0x7fedf81606c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf7fff700 1 -- 172.21.15.113:0/1571218845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec009550 con 0x7fedf8071b50 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf7fff700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedf8071b50 0x7fedf80cdb90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fede800c460 tx=0x7fede800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf4ff9700 1 -- 172.21.15.113:0/1571218845 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede80052d0 con 0x7fedf8071b50 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf4ff9700 1 -- 172.21.15.113:0/1571218845 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede8005450 con 0x7fedf8071b50 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedfe98c700 1 -- 172.21.15.113:0/1571218845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf815e620 con 0x7fedf8071b50 2024-01-15T11:27:02.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedf4ff9700 1 -- 172.21.15.113:0/1571218845 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8006aa0 con 0x7fedf8071b50 2024-01-15T11:27:02.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.836+0000 7fedfe98c700 1 -- 172.21.15.113:0/1571218845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf80cce80 con 0x7fedf8071b50 2024-01-15T11:27:02.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.837+0000 7fedfe98c700 1 -- 172.21.15.113:0/1571218845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf8067f60 con 0x7fedf8071b50 2024-01-15T11:27:02.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.838+0000 7fedf4ff9700 1 -- 172.21.15.113:0/1571218845 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fede8003680 con 0x7fedf8071b50 2024-01-15T11:27:02.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.838+0000 7fedf4ff9700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fede406b500 0x7fede406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:02.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.838+0000 7fedf4ff9700 1 -- 172.21.15.113:0/1571218845 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fede8089060 con 0x7fedf8071b50 2024-01-15T11:27:02.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.838+0000 7fedf77fe700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fede406b500 0x7fede406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:02.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.839+0000 7fedf77fe700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fede406b500 0x7fede406d9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fedf815b010 tx=0x7fedec008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:02.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:02.843+0000 7fedf4ff9700 1 -- 172.21.15.113:0/1571218845 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fede8053ce0 con 0x7fedf8071b50 2024-01-15T11:27:03.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.070+0000 7fedfe98c700 1 -- 172.21.15.113:0/1571218845 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fedf80cb060 con 0x7fede406b500 2024-01-15T11:27:03.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.086+0000 7fedf4ff9700 1 -- 172.21.15.113:0/1571218845 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fedf80cb060 con 0x7fede406b500 2024-01-15T11:27:03.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.088+0000 7fedfe98c700 1 -- 172.21.15.113:0/1571218845 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fede406b500 msgr2=0x7fede406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:03.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.088+0000 7fedfe98c700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fede406b500 0x7fede406d9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fedf815b010 tx=0x7fedec008030 comp rx=0 tx=0).stop 2024-01-15T11:27:03.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.088+0000 7fedfe98c700 1 -- 172.21.15.113:0/1571218845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedf8071b50 msgr2=0x7fedf80cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:03.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.088+0000 7fedfe98c700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedf8071b50 0x7fedf80cdb90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fede800c460 tx=0x7fede800c820 comp rx=0 tx=0).stop 2024-01-15T11:27:03.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.088+0000 7fedfe98c700 1 -- 172.21.15.113:0/1571218845 shutdown_connections 2024-01-15T11:27:03.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.088+0000 7fedfe98c700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fede406b500 0x7fede406d9c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:03.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.088+0000 7fedfe98c700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fedf815e0c0 0x7fedf80cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:03.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.088+0000 7fedfe98c700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedf8071b50 0x7fedf80cdb90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:03.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.088+0000 7fedfe98c700 1 --2- 172.21.15.113:0/1571218845 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fedf80d3a40 0x7fedf81606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:03.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.088+0000 7fedfe98c700 1 -- 172.21.15.113:0/1571218845 >> 172.21.15.113:0/1571218845 conn(0x7fedf806b620 msgr2=0x7fedf81553d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:03.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.089+0000 7fedfe98c700 1 -- 172.21.15.113:0/1571218845 shutdown_connections 2024-01-15T11:27:03.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.089+0000 7fedfe98c700 1 -- 172.21.15.113:0/1571218845 wait complete. 2024-01-15T11:27:03.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:27:03.104 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T11:27:03.141 INFO:teuthology.orchestra.run.smithi113.stdout:2 2024-01-15T11:27:03.141 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T11:27:03.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.306+0000 7fd9ac090700 1 Processor -- start 2024-01-15T11:27:03.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.306+0000 7fd9ac090700 1 -- start start 2024-01-15T11:27:03.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.307+0000 7fd9ac090700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fd9a40ae750 0x7fd9a406e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:03.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.307+0000 7fd9ac090700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a40b4940 con 0x7fd9a406c710 2024-01-15T11:27:03.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.307+0000 7fd9ac090700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a40b4c20 con 0x7fd9a4147840 2024-01-15T11:27:03.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.307+0000 7fd9ac090700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a40b4f00 con 0x7fd9a40ae750 2024-01-15T11:27:03.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.307+0000 7fd9a962b700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fd9a406c710 0x7fd9a406d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51940/0 (socket says 172.21.15.113:51940) 2024-01-15T11:27:03.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.307+0000 7fd9a962b700 1 -- 172.21.15.113:0/3981172055 learned_addr learned my addr 172.21.15.113:0/3981172055 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:27:03.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.307+0000 7fd9aa62d700 1 --2- 172.21.15.113:0/3981172055 >> v2:172.21.15.141:3300/0 conn(0x7fd9a40ae750 0x7fd9a406e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:03.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.307+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186284972 0 0) 0x7fd9a40b4940 con 0x7fd9a406c710 2024-01-15T11:27:03.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.307+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9900035b0 con 0x7fd9a406c710 2024-01-15T11:27:03.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3406729403 0 0) 0x7fd9900035b0 con 0x7fd9a406c710 2024-01-15T11:27:03.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 >> v2:172.21.15.141:3300/0 conn(0x7fd9a40ae750 msgr2=0x7fd9a406e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:03.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 --2- 172.21.15.113:0/3981172055 >> v2:172.21.15.141:3300/0 conn(0x7fd9a40ae750 0x7fd9a406e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 >> v1:172.21.15.141:6789/0 conn(0x7fd9a4147840 legacy=0x7fd9a406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a40b51e0 con 0x7fd9a406c710 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1661700084 0 0) 0x7fd9a0001670 con 0x7fd9a406c710 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 >> v1:172.21.15.113:6789/0 conn(0x7fd9a406c710 legacy=0x7fd9a406d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 --2- 172.21.15.113:0/3981172055 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd990003a00 0x7fd990003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 --2- 172.21.15.113:0/3981172055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd990004850 0x7fd990004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 --2- 172.21.15.113:0/3981172055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd990005210 0x7fd99000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a40b4f00 con 0x7fd990004850 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99000c0b0 con 0x7fd990005210 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99000c270 con 0x7fd990003a00 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a962b700 1 --2- 172.21.15.113:0/3981172055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd990004850 0x7fd990004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9aa62d700 1 --2- 172.21.15.113:0/3981172055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd990005210 0x7fd99000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a9e2c700 1 --2- 172.21.15.113:0/3981172055 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd990003a00 0x7fd990003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a962b700 1 -- 172.21.15.113:0/3981172055 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd990003a00 msgr2=0x7fd990003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:03.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a962b700 1 --2- 172.21.15.113:0/3981172055 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd990003a00 0x7fd990003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a962b700 1 -- 172.21.15.113:0/3981172055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd990005210 msgr2=0x7fd99000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a962b700 1 --2- 172.21.15.113:0/3981172055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd990005210 0x7fd99000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a962b700 1 -- 172.21.15.113:0/3981172055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9900035b0 con 0x7fd990004850 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9aa62d700 1 --2- 172.21.15.113:0/3981172055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd990005210 0x7fd99000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9aa62d700 1 -- 172.21.15.113:0/3981172055 reap_dead start 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a962b700 1 --2- 172.21.15.113:0/3981172055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd990004850 0x7fd990004cd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000c240 tx=0x7fd990005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9534b9ca1c5276fb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0007c10 con 0x7fd990004850 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a000cde0 con 0x7fd990004850 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.308+0000 7fd9a8e2a700 1 -- 172.21.15.113:0/3981172055 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0005ba0 con 0x7fd990004850 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.309+0000 7fd9ac090700 1 -- 172.21.15.113:0/3981172055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd990004850 msgr2=0x7fd990004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.309+0000 7fd9ac090700 1 --2- 172.21.15.113:0/3981172055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd990004850 0x7fd990004cd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000c240 tx=0x7fd990005880 comp rx=0 tx=0).stop 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.309+0000 7fd9ac090700 1 -- 172.21.15.113:0/3981172055 shutdown_connections 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.309+0000 7fd9ac090700 1 --2- 172.21.15.113:0/3981172055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd990004850 0x7fd990004cd0 secure :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000c240 tx=0x7fd990005880 comp rx=0 tx=0).stop 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.309+0000 7fd9ac090700 1 -- 172.21.15.113:0/3981172055 >> 172.21.15.113:0/3981172055 conn(0x7fd9a4071c70 msgr2=0x7fd9a4072050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:03.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.309+0000 7fd9ac090700 1 -- 172.21.15.113:0/3981172055 shutdown_connections 2024-01-15T11:27:03.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.309+0000 7fd9ac090700 1 -- 172.21.15.113:0/3981172055 wait complete. 2024-01-15T11:27:03.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.310+0000 7fd9ac090700 1 Processor -- start 2024-01-15T11:27:03.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.310+0000 7fd9ac090700 1 -- start start 2024-01-15T11:27:03.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.311+0000 7fd9ac090700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9a415d7e0 0x7fd9a4143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:03.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.311+0000 7fd9ac090700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9a4144480 0x7fd9a41e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:03.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.311+0000 7fd9ac090700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9a41e5040 0x7fd9a41e3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:03.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.311+0000 7fd9ac090700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a41e54d0 con 0x7fd9a41e5040 2024-01-15T11:27:03.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.311+0000 7fd9ac090700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a41432e0 con 0x7fd9a4144480 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.311+0000 7fd9ac090700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a41426d0 con 0x7fd9a415d7e0 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.311+0000 7fd9aa62d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9a41e5040 0x7fd9a41e3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.311+0000 7fd9aa62d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9a41e5040 0x7fd9a41e3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35110/0 (socket says 172.21.15.113:35110) 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.311+0000 7fd9aa62d700 1 -- 172.21.15.113:0/520662554 learned_addr learned my addr 172.21.15.113:0/520662554 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.311+0000 7fd9a962b700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9a4144480 0x7fd9a41e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.311+0000 7fd9a9e2c700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9a415d7e0 0x7fd9a4143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.312+0000 7fd9aa62d700 1 -- 172.21.15.113:0/520662554 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9a415d7e0 msgr2=0x7fd9a4143f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.312+0000 7fd9aa62d700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9a415d7e0 0x7fd9a4143f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.312+0000 7fd9aa62d700 1 -- 172.21.15.113:0/520662554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9a4144480 msgr2=0x7fd9a41e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.312+0000 7fd9aa62d700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9a4144480 0x7fd9a41e4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.312+0000 7fd9aa62d700 1 -- 172.21.15.113:0/520662554 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a41429b0 con 0x7fd9a41e5040 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.312+0000 7fd9aa62d700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9a41e5040 0x7fd9a41e3010 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40b4320 tx=0x7fd99c007f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.312+0000 7fd99a7fc700 1 -- 172.21.15.113:0/520662554 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c01b070 con 0x7fd9a41e5040 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.312+0000 7fd99a7fc700 1 -- 172.21.15.113:0/520662554 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c00f040 con 0x7fd9a41e5040 2024-01-15T11:27:03.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.312+0000 7fd9ac090700 1 -- 172.21.15.113:0/520662554 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a4142c90 con 0x7fd9a41e5040 2024-01-15T11:27:03.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.312+0000 7fd99a7fc700 1 -- 172.21.15.113:0/520662554 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c00e050 con 0x7fd9a41e5040 2024-01-15T11:27:03.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.312+0000 7fd9ac090700 1 -- 172.21.15.113:0/520662554 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a4142380 con 0x7fd9a41e5040 2024-01-15T11:27:03.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.313+0000 7fd9ac090700 1 -- 172.21.15.113:0/520662554 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a4143470 con 0x7fd9a41e5040 2024-01-15T11:27:03.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.313+0000 7fd99a7fc700 1 -- 172.21.15.113:0/520662554 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fd99c010930 con 0x7fd9a41e5040 2024-01-15T11:27:03.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.314+0000 7fd99a7fc700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd98806b500 0x7fd98806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:03.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.314+0000 7fd99a7fc700 1 -- 172.21.15.113:0/520662554 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd99c08a010 con 0x7fd9a41e5040 2024-01-15T11:27:03.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.314+0000 7fd9a9e2c700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd98806b500 0x7fd98806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:03.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.314+0000 7fd9a9e2c700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd98806b500 0x7fd98806d9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd994000c00 tx=0x7fd994009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:03.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.318+0000 7fd99a7fc700 1 -- 172.21.15.113:0/520662554 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd99c054af0 con 0x7fd9a41e5040 2024-01-15T11:27:03.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.458+0000 7fd99a7fc700 1 -- 172.21.15.113:0/520662554 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 8) v1 ==== 89794+0+0 (secure 0 0 0) 0x7fd99c054ec0 con 0x7fd9a41e5040 2024-01-15T11:27:03.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:03.539+0000 7fd9ac090700 1 -- 172.21.15.113:0/520662554 --> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9a40b02b0 con 0x7fd98806b500 2024-01-15T11:27:03.722 INFO:tasks.ceph.mon.a.smithi113.stderr:2024-01-15T11:27:03.723+0000 7f80ebd4a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T11:27:03.781 INFO:tasks.ceph.mds.a.smithi113.stdout:starting mds.a at 2024-01-15T11:27:05.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:05.750+0000 7fd99a7fc700 1 -- 172.21.15.113:0/520662554 <== mon.0 v2:172.21.15.113:3300/0 8 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fd99c08c550 con 0x7fd9a41e5040 2024-01-15T11:27:06.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.770+0000 7fd99a7fc700 1 -- 172.21.15.113:0/520662554 <== mgr.4104 v2:172.21.15.113:6832/3789331922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fd994015520 con 0x7fd98806b500 2024-01-15T11:27:06.770 INFO:teuthology.orchestra.run.smithi113.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T11:27:06.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.778+0000 7fd9ac090700 1 -- 172.21.15.113:0/520662554 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd98806b500 msgr2=0x7fd98806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:06.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.778+0000 7fd9ac090700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd98806b500 0x7fd98806d9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd994000c00 tx=0x7fd994009040 comp rx=0 tx=0).stop 2024-01-15T11:27:06.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.778+0000 7fd9ac090700 1 -- 172.21.15.113:0/520662554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9a41e5040 msgr2=0x7fd9a41e3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:06.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.778+0000 7fd9ac090700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9a41e5040 0x7fd9a41e3010 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40b4320 tx=0x7fd99c007f20 comp rx=0 tx=0).stop 2024-01-15T11:27:06.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.778+0000 7fd9ac090700 1 -- 172.21.15.113:0/520662554 shutdown_connections 2024-01-15T11:27:06.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.778+0000 7fd9ac090700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fd98806b500 0x7fd98806d9c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:06.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.778+0000 7fd9ac090700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9a41e5040 0x7fd9a41e3010 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:06.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.778+0000 7fd9ac090700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fd9a415d7e0 0x7fd9a4143f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:06.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.778+0000 7fd9ac090700 1 --2- 172.21.15.113:0/520662554 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd9a4144480 0x7fd9a41e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:06.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.778+0000 7fd9ac090700 1 -- 172.21.15.113:0/520662554 >> 172.21.15.113:0/520662554 conn(0x7fd9a4071c70 msgr2=0x7fd9a406d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:06.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.778+0000 7fd9ac090700 1 -- 172.21.15.113:0/520662554 shutdown_connections 2024-01-15T11:27:06.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:06.779+0000 7fd9ac090700 1 -- 172.21.15.113:0/520662554 wait complete. 2024-01-15T11:27:06.801 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T11:27:06.801 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T11:27:06.872 DEBUG:teuthology.orchestra.run.smithi141:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T11:27:06.975 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T11:27:06.976 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:27:06.976 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:27:07.009 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-15T11:27:07.009 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T11:27:07.070 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T11:27:07.232 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.232+0000 7fc1c8598700 1 Processor -- start 2024-01-15T11:27:07.232 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.232+0000 7fc1c8598700 1 -- start start 2024-01-15T11:27:07.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.233+0000 7fc1c8598700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc1c015af10 0x7fc1c015e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.233+0000 7fc1c8598700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1c00ce140 0x7fc1c015ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.233+0000 7fc1c8598700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c01414c0 con 0x7fc1c015f460 2024-01-15T11:27:07.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.233+0000 7fc1c8598700 1 -- --> v2:172.21.15.141:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c015f8c0 con 0x7fc1c015af10 2024-01-15T11:27:07.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.233+0000 7fc1c8598700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00ce9f0 con 0x7fc1c00ce140 2024-01-15T11:27:07.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c6334700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc1c015af10 0x7fc1c015e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c5b33700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1c00ce140 0x7fc1c015ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c6334700 1 --2- >> v2:172.21.15.141:3300/0 conn(0x7fc1c015af10 0x7fc1c015e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.113:50072/0 (socket says 172.21.15.113:50072) 2024-01-15T11:27:07.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c6b35700 1 --1- >> v1:172.21.15.141:6789/0 conn(0x7fc1c015f460 0x7fc1c015b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.141:6789/0 says I am v1:172.21.15.113:40602/0 (socket says 172.21.15.113:40602) 2024-01-15T11:27:07.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c6334700 1 -- 172.21.15.113:0/2856862525 learned_addr learned my addr 172.21.15.113:0/2856862525 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:07.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c5b33700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1c00ce140 0x7fc1c015ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3301/0 says I am v2:172.21.15.113:55788/0 (socket says 172.21.15.113:55788) 2024-01-15T11:27:07.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c6334700 1 -- 172.21.15.113:0/2856862525 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1c00ce140 msgr2=0x7fc1c015ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c6334700 1 --2- 172.21.15.113:0/2856862525 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1c00ce140 0x7fc1c015ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c6334700 1 -- 172.21.15.113:0/2856862525 >> v1:172.21.15.141:6789/0 conn(0x7fc1c015f460 legacy=0x7fc1c015b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c6334700 1 -- 172.21.15.113:0/2856862525 --> v2:172.21.15.141:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c00cecd0 con 0x7fc1c015af10 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c5332700 1 -- 172.21.15.113:0/2856862525 <== mon.1 v1:172.21.15.141:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189990064 0 0) 0x7fc1c01414c0 con 0x7fc1c015f460 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c5b33700 1 --2- 172.21.15.113:0/2856862525 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1c00ce140 0x7fc1c015ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c6334700 1 --2- 172.21.15.113:0/2856862525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1c015af10 0x7fc1c015e9e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc1b40098f0 tx=0x7fc1b4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94f10c1866e06ce7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c5332700 1 -- 172.21.15.113:0/2856862525 <== mon.1 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4011830 con 0x7fc1c015af10 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c5332700 1 -- 172.21.15.113:0/2856862525 <== mon.1 v2:172.21.15.141:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b4011990 con 0x7fc1c015af10 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.234+0000 7fc1c5332700 1 -- 172.21.15.113:0/2856862525 <== mon.1 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4011d20 con 0x7fc1c015af10 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.235+0000 7fc1c8598700 1 -- 172.21.15.113:0/2856862525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1c015af10 msgr2=0x7fc1c015e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.235+0000 7fc1c8598700 1 --2- 172.21.15.113:0/2856862525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1c015af10 0x7fc1c015e9e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc1b40098f0 tx=0x7fc1b4009c00 comp rx=0 tx=0).stop 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.235+0000 7fc1c8598700 1 -- 172.21.15.113:0/2856862525 shutdown_connections 2024-01-15T11:27:07.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.235+0000 7fc1c8598700 1 --2- 172.21.15.113:0/2856862525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1c015af10 0x7fc1c015e9e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.235+0000 7fc1c8598700 1 --2- 172.21.15.113:0/2856862525 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1c00ce140 0x7fc1c015ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.235+0000 7fc1c8598700 1 -- 172.21.15.113:0/2856862525 >> 172.21.15.113:0/2856862525 conn(0x7fc1c00cd570 msgr2=0x7fc1c00cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:07.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.235+0000 7fc1c8598700 1 -- 172.21.15.113:0/2856862525 shutdown_connections 2024-01-15T11:27:07.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.235+0000 7fc1c8598700 1 -- 172.21.15.113:0/2856862525 wait complete. 2024-01-15T11:27:07.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.236+0000 7fc1c8598700 1 Processor -- start 2024-01-15T11:27:07.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.236+0000 7fc1c8598700 1 -- start start 2024-01-15T11:27:07.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c8598700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1c00ce140 0x7fc1c013d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c8598700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1c015af10 0x7fc1c0158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c8598700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1c015f460 0x7fc1c015cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c8598700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c01f58a0 con 0x7fc1c015af10 2024-01-15T11:27:07.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c8598700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c015a5c0 con 0x7fc1c015f460 2024-01-15T11:27:07.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c8598700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c01f5af0 con 0x7fc1c00ce140 2024-01-15T11:27:07.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c5b33700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1c015af10 0x7fc1c0158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c5b33700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1c015af10 0x7fc1c0158710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42144/0 (socket says 172.21.15.113:42144) 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c5b33700 1 -- 172.21.15.113:0/720397734 learned_addr learned my addr 172.21.15.113:0/720397734 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c6b35700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1c015f460 0x7fc1c015cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c6334700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1c00ce140 0x7fc1c013d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c5b33700 1 -- 172.21.15.113:0/720397734 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1c00ce140 msgr2=0x7fc1c013d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c5b33700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1c00ce140 0x7fc1c013d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c5b33700 1 -- 172.21.15.113:0/720397734 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1c015f460 msgr2=0x7fc1c015cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c5b33700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1c015f460 0x7fc1c015cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c5b33700 1 -- 172.21.15.113:0/720397734 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc0094a0 con 0x7fc1c015af10 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.237+0000 7fc1c5b33700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1c015af10 0x7fc1c0158710 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00ebe0 tx=0x7fc1bc009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.238+0000 7fc1b2ffd700 1 -- 172.21.15.113:0/720397734 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc00c890 con 0x7fc1c015af10 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.238+0000 7fc1c8598700 1 -- 172.21.15.113:0/720397734 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b4009550 con 0x7fc1c015af10 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.238+0000 7fc1b2ffd700 1 -- 172.21.15.113:0/720397734 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1bc00c9f0 con 0x7fc1c015af10 2024-01-15T11:27:07.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.238+0000 7fc1b2ffd700 1 -- 172.21.15.113:0/720397734 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc00cda0 con 0x7fc1c015af10 2024-01-15T11:27:07.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.238+0000 7fc1c8598700 1 -- 172.21.15.113:0/720397734 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c01f60b0 con 0x7fc1c015af10 2024-01-15T11:27:07.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.239+0000 7fc1c8598700 1 -- 172.21.15.113:0/720397734 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c0067f60 con 0x7fc1c015af10 2024-01-15T11:27:07.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.239+0000 7fc1b2ffd700 1 -- 172.21.15.113:0/720397734 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc1bc013470 con 0x7fc1c015af10 2024-01-15T11:27:07.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.240+0000 7fc1b2ffd700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc1a406b590 0x7fc1a406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.240+0000 7fc1c6334700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc1a406b590 0x7fc1a406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.240+0000 7fc1b2ffd700 1 -- 172.21.15.113:0/720397734 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc1bc089b40 con 0x7fc1c015af10 2024-01-15T11:27:07.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.240+0000 7fc1c6334700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc1a406b590 0x7fc1a406da50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4000c00 tx=0x7fc1b4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:07.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.244+0000 7fc1b2ffd700 1 -- 172.21.15.113:0/720397734 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1bc054940 con 0x7fc1c015af10 2024-01-15T11:27:07.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.437+0000 7fc1c8598700 1 -- 172.21.15.113:0/720397734 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1c00ce6d0 con 0x7fc1c015af10 2024-01-15T11:27:07.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.438+0000 7fc1b2ffd700 1 -- 172.21.15.113:0/720397734 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+13089 (secure 0 0 0) 0x7fc1bc057f60 con 0x7fc1c015af10 2024-01-15T11:27:07.438 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-15T11:27:07.438 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":36,"fsid":"910f8628-cd75-4a37-bf4e-e3603a95290d","created":"2024-01-15T11:24:50.433558+0000","modified":"2024-01-15T11:27:06.756678+0000","last_up_change":"2024-01-15T11:25:01.492840+0000","last_in_change":"2024-01-15T11:24:52.950196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T11:25:02.149657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"04532dcb-3e36-440a-a9c1-c0d2158ab5e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6801","nonce":1061671788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6803","nonce":1061671788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6807","nonce":1061671788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1061671788},{"type":"v1","addr":"172.21.15.113:6805","nonce":1061671788}]},"public_addr":"172.21.15.113:6801/1061671788","cluster_addr":"172.21.15.113:6803/1061671788","heartbeat_back_addr":"172.21.15.113:6807/1061671788","heartbeat_front_addr":"172.21.15.113:6805/1061671788","state":["exists","up"]},{"osd":1,"uuid":"17b10b62-1681-487b-87ad-36a5e5e4bb57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6825","nonce":2814390340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6827","nonce":2814390340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6831","nonce":2814390340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2814390340},{"type":"v1","addr":"172.21.15.113:6829","nonce":2814390340}]},"public_addr":"172.21.15.113:6825/2814390340","cluster_addr":"172.21.15.113:6827/2814390340","heartbeat_back_addr":"172.21.15.113:6831/2814390340","heartbeat_front_addr":"172.21.15.113:6829/2814390340","state":["exists","up"]},{"osd":2,"uuid":"1c190417-61f7-4b0b-888a-cc8bd7c03678","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6817","nonce":1186622624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6819","nonce":1186622624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6823","nonce":1186622624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1186622624},{"type":"v1","addr":"172.21.15.113:6821","nonce":1186622624}]},"public_addr":"172.21.15.113:6817/1186622624","cluster_addr":"172.21.15.113:6819/1186622624","heartbeat_back_addr":"172.21.15.113:6823/1186622624","heartbeat_front_addr":"172.21.15.113:6821/1186622624","state":["exists","up"]},{"osd":3,"uuid":"7f26b85c-4aac-4d47-b9dd-8872d5b5cc11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6809","nonce":2024634597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6811","nonce":2024634597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6815","nonce":2024634597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2024634597},{"type":"v1","addr":"172.21.15.113:6813","nonce":2024634597}]},"public_addr":"172.21.15.113:6809/2024634597","cluster_addr":"172.21.15.113:6811/2024634597","heartbeat_back_addr":"172.21.15.113:6815/2024634597","heartbeat_front_addr":"172.21.15.113:6813/2024634597","state":["exists","up"]},{"osd":4,"uuid":"c44cd675-0510-468f-9e5f-4388ba1c00bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6801","nonce":1209001813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6803","nonce":1209001813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6807","nonce":1209001813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1209001813},{"type":"v1","addr":"172.21.15.141:6805","nonce":1209001813}]},"public_addr":"172.21.15.141:6801/1209001813","cluster_addr":"172.21.15.141:6803/1209001813","heartbeat_back_addr":"172.21.15.141:6807/1209001813","heartbeat_front_addr":"172.21.15.141:6805/1209001813","state":["exists","up"]},{"osd":5,"uuid":"419abe5e-bcf4-4bd1-8c62-cf481293a40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6809","nonce":1489198687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6811","nonce":1489198687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6815","nonce":1489198687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1489198687},{"type":"v1","addr":"172.21.15.141:6813","nonce":1489198687}]},"public_addr":"172.21.15.141:6809/1489198687","cluster_addr":"172.21.15.141:6811/1489198687","heartbeat_back_addr":"172.21.15.141:6815/1489198687","heartbeat_front_addr":"172.21.15.141:6813/1489198687","state":["exists","up"]},{"osd":6,"uuid":"9cc1e986-5a2c-4854-b5ac-3eabdea3e7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6825","nonce":3801965007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6827","nonce":3801965007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6831","nonce":3801965007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":3801965007},{"type":"v1","addr":"172.21.15.141:6829","nonce":3801965007}]},"public_addr":"172.21.15.141:6825/3801965007","cluster_addr":"172.21.15.141:6827/3801965007","heartbeat_back_addr":"172.21.15.141:6831/3801965007","heartbeat_front_addr":"172.21.15.141:6829/3801965007","state":["exists","up"]},{"osd":7,"uuid":"5b587b01-e25d-47a8-8bd7-2ec6803cedd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6817","nonce":2034940459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6819","nonce":2034940459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6823","nonce":2034940459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":2034940459},{"type":"v1","addr":"172.21.15.141:6821","nonce":2034940459}]},"public_addr":"172.21.15.141:6817/2034940459","cluster_addr":"172.21.15.141:6819/2034940459","heartbeat_back_addr":"172.21.15.141:6823/2034940459","heartbeat_front_addr":"172.21.15.141:6821/2034940459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.094947+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.137210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.165714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.148409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.180186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.183430+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.161645+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T11:24:58.113410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.113:6837/436972884":"2024-01-16T11:27:03.723719+0000","172.21.15.113:6835/436972884":"2024-01-16T11:27:03.723719+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T11:27:07.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.440+0000 7fc1c8598700 1 -- 172.21.15.113:0/720397734 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc1a406b590 msgr2=0x7fc1a406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.440+0000 7fc1c8598700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc1a406b590 0x7fc1a406da50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4000c00 tx=0x7fc1b4008030 comp rx=0 tx=0).stop 2024-01-15T11:27:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.440+0000 7fc1c8598700 1 -- 172.21.15.113:0/720397734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1c015af10 msgr2=0x7fc1c0158710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.440+0000 7fc1c8598700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1c015af10 0x7fc1c0158710 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00ebe0 tx=0x7fc1bc009e60 comp rx=0 tx=0).stop 2024-01-15T11:27:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.440+0000 7fc1c8598700 1 -- 172.21.15.113:0/720397734 shutdown_connections 2024-01-15T11:27:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.440+0000 7fc1c8598700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7fc1a406b590 0x7fc1a406da50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.440+0000 7fc1c8598700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc1c015f460 0x7fc1c015cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.440+0000 7fc1c8598700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7fc1c00ce140 0x7fc1c013d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.440+0000 7fc1c8598700 1 --2- 172.21.15.113:0/720397734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1c015af10 0x7fc1c0158710 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.440+0000 7fc1c8598700 1 -- 172.21.15.113:0/720397734 >> 172.21.15.113:0/720397734 conn(0x7fc1c00cd570 msgr2=0x7fc1c015c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.440+0000 7fc1c8598700 1 -- 172.21.15.113:0/720397734 shutdown_connections 2024-01-15T11:27:07.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.440+0000 7fc1c8598700 1 -- 172.21.15.113:0/720397734 wait complete. 2024-01-15T11:27:07.453 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T11:25:02.149657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T11:27:07.453 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T11:27:07.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.579+0000 7f9c63d0b700 1 Processor -- start 2024-01-15T11:27:07.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.579+0000 7f9c63d0b700 1 -- start start 2024-01-15T11:27:07.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.580+0000 7f9c63d0b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9c5c161690 0x7f9c5c0d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.580+0000 7f9c63d0b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9c5c09b110 0x7f9c5c0d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.580+0000 7f9c63d0b700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c0d0a70 con 0x7f9c5c161a70 2024-01-15T11:27:07.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.580+0000 7f9c63d0b700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c5c0d1c10 con 0x7f9c5c161690 2024-01-15T11:27:07.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.580+0000 7f9c63d0b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c0b2ab0 con 0x7f9c5c09b110 2024-01-15T11:27:07.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.580+0000 7f9c61aa7700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9c5c161690 0x7f9c5c0d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.580+0000 7f9c61aa7700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9c5c161690 0x7f9c5c0d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42150/0 (socket says 172.21.15.113:42150) 2024-01-15T11:27:07.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.580+0000 7f9c61aa7700 1 -- 172.21.15.113:0/660628195 learned_addr learned my addr 172.21.15.113:0/660628195 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:07.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.581+0000 7f9c622a8700 1 --2- 172.21.15.113:0/660628195 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9c5c09b110 0x7f9c5c0d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.581+0000 7f9c61aa7700 1 -- 172.21.15.113:0/660628195 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9c5c09b110 msgr2=0x7f9c5c0d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:07.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.581+0000 7f9c61aa7700 1 --2- 172.21.15.113:0/660628195 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9c5c09b110 0x7f9c5c0d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.581+0000 7f9c61aa7700 1 -- 172.21.15.113:0/660628195 >> v1:172.21.15.141:6789/0 conn(0x7f9c5c161a70 legacy=0x7f9c5c09aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.581+0000 7f9c61aa7700 1 -- 172.21.15.113:0/660628195 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c0b2d90 con 0x7f9c5c161690 2024-01-15T11:27:07.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.581+0000 7f9c61aa7700 1 --2- 172.21.15.113:0/660628195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c5c161690 0x7f9c5c0d0f20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9c580098f0 tx=0x7f9c58009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2caaac743fbe6b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:07.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.581+0000 7f9c60aa5700 1 -- 172.21.15.113:0/660628195 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5800b8a0 con 0x7f9c5c161690 2024-01-15T11:27:07.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.581+0000 7f9c60aa5700 1 -- 172.21.15.113:0/660628195 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5800ba00 con 0x7f9c5c161690 2024-01-15T11:27:07.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.581+0000 7f9c60aa5700 1 -- 172.21.15.113:0/660628195 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5800bd90 con 0x7f9c5c161690 2024-01-15T11:27:07.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.582+0000 7f9c63d0b700 1 -- 172.21.15.113:0/660628195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c5c161690 msgr2=0x7f9c5c0d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.582+0000 7f9c63d0b700 1 --2- 172.21.15.113:0/660628195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c5c161690 0x7f9c5c0d0f20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9c580098f0 tx=0x7f9c58009c00 comp rx=0 tx=0).stop 2024-01-15T11:27:07.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.582+0000 7f9c63d0b700 1 -- 172.21.15.113:0/660628195 shutdown_connections 2024-01-15T11:27:07.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.582+0000 7f9c63d0b700 1 --2- 172.21.15.113:0/660628195 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9c5c09b110 0x7f9c5c0d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.582+0000 7f9c63d0b700 1 --2- 172.21.15.113:0/660628195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c5c161690 0x7f9c5c0d0f20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.582+0000 7f9c63d0b700 1 -- 172.21.15.113:0/660628195 >> 172.21.15.113:0/660628195 conn(0x7f9c5c0d1e50 msgr2=0x7f9c5c0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:07.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.582+0000 7f9c63d0b700 1 -- 172.21.15.113:0/660628195 shutdown_connections 2024-01-15T11:27:07.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.582+0000 7f9c63d0b700 1 -- 172.21.15.113:0/660628195 wait complete. 2024-01-15T11:27:07.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.583+0000 7f9c63d0b700 1 Processor -- start 2024-01-15T11:27:07.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.583+0000 7f9c63d0b700 1 -- start start 2024-01-15T11:27:07.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c63d0b700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9c5c09b110 0x7f9c5c0c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c63d0b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c5c161690 0x7f9c5c1470a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c63d0b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c5c161a70 0x7f9c5c14add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c63d0b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c15efd0 con 0x7f9c5c161690 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c63d0b700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c1475e0 con 0x7f9c5c161a70 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c63d0b700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c147880 con 0x7f9c5c09b110 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c612a6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c5c161690 0x7f9c5c1470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c612a6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c5c161690 0x7f9c5c1470a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42152/0 (socket says 172.21.15.113:42152) 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c612a6700 1 -- 172.21.15.113:0/3388825878 learned_addr learned my addr 172.21.15.113:0/3388825878 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c61aa7700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9c5c09b110 0x7f9c5c0c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c622a8700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c5c161a70 0x7f9c5c14add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c612a6700 1 -- 172.21.15.113:0/3388825878 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9c5c09b110 msgr2=0x7f9c5c0c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c612a6700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9c5c09b110 0x7f9c5c0c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c612a6700 1 -- 172.21.15.113:0/3388825878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c5c161a70 msgr2=0x7f9c5c14add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c612a6700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c5c161a70 0x7f9c5c14add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c612a6700 1 -- 172.21.15.113:0/3388825878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c58009550 con 0x7f9c5c161690 2024-01-15T11:27:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c612a6700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c5c161690 0x7f9c5c1470a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00c450 tx=0x7f9c4c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:07.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c527fc700 1 -- 172.21.15.113:0/3388825878 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c0052d0 con 0x7f9c5c161690 2024-01-15T11:27:07.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c527fc700 1 -- 172.21.15.113:0/3388825878 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4c005430 con 0x7f9c5c161690 2024-01-15T11:27:07.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c63d0b700 1 -- 172.21.15.113:0/3388825878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5c147b60 con 0x7f9c5c161690 2024-01-15T11:27:07.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c527fc700 1 -- 172.21.15.113:0/3388825878 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c0038c0 con 0x7f9c5c161690 2024-01-15T11:27:07.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.584+0000 7f9c63d0b700 1 -- 172.21.15.113:0/3388825878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5c15f1e0 con 0x7f9c5c161690 2024-01-15T11:27:07.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.585+0000 7f9c63d0b700 1 -- 172.21.15.113:0/3388825878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5c0d1710 con 0x7f9c5c161690 2024-01-15T11:27:07.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.586+0000 7f9c527fc700 1 -- 172.21.15.113:0/3388825878 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f9c4c003a20 con 0x7f9c5c161690 2024-01-15T11:27:07.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.586+0000 7f9c527fc700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f9c4006b590 0x7f9c4006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.587+0000 7f9c61aa7700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f9c4006b590 0x7f9c4006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.587+0000 7f9c527fc700 1 -- 172.21.15.113:0/3388825878 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9c4c088f20 con 0x7f9c5c161690 2024-01-15T11:27:07.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.587+0000 7f9c61aa7700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f9c4006b590 0x7f9c4006da50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009c40 tx=0x7f9c58008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:07.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.591+0000 7f9c527fc700 1 -- 172.21.15.113:0/3388825878 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c5c0d1710 con 0x7f9c5c161690 2024-01-15T11:27:07.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.781+0000 7f9c63d0b700 1 -- 172.21.15.113:0/3388825878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c5c0d1710 con 0x7f9c5c161690 2024-01-15T11:27:07.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.781+0000 7f9c527fc700 1 -- 172.21.15.113:0/3388825878 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c5c0d1710 con 0x7f9c5c161690 2024-01-15T11:27:07.781 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-15T11:27:07.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.783+0000 7f9c63d0b700 1 -- 172.21.15.113:0/3388825878 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f9c4006b590 msgr2=0x7f9c4006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.783+0000 7f9c63d0b700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f9c4006b590 0x7f9c4006da50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009c40 tx=0x7f9c58008030 comp rx=0 tx=0).stop 2024-01-15T11:27:07.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.783+0000 7f9c63d0b700 1 -- 172.21.15.113:0/3388825878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c5c161690 msgr2=0x7f9c5c1470a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.783+0000 7f9c63d0b700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c5c161690 0x7f9c5c1470a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00c450 tx=0x7f9c4c00c760 comp rx=0 tx=0).stop 2024-01-15T11:27:07.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.783+0000 7f9c63d0b700 1 -- 172.21.15.113:0/3388825878 shutdown_connections 2024-01-15T11:27:07.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.783+0000 7f9c63d0b700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f9c4006b590 0x7f9c4006da50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.783+0000 7f9c63d0b700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9c5c161a70 0x7f9c5c14add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.783+0000 7f9c63d0b700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f9c5c09b110 0x7f9c5c0c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.783+0000 7f9c63d0b700 1 --2- 172.21.15.113:0/3388825878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c5c161690 0x7f9c5c1470a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.783+0000 7f9c63d0b700 1 -- 172.21.15.113:0/3388825878 >> 172.21.15.113:0/3388825878 conn(0x7f9c5c0d1e50 msgr2=0x7f9c5c061a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:07.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.783+0000 7f9c63d0b700 1 -- 172.21.15.113:0/3388825878 shutdown_connections 2024-01-15T11:27:07.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.783+0000 7f9c63d0b700 1 -- 172.21.15.113:0/3388825878 wait complete. 2024-01-15T11:27:07.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T11:27:07.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.918+0000 7f61577c7700 1 Processor -- start 2024-01-15T11:27:07.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.919+0000 7f61577c7700 1 -- start start 2024-01-15T11:27:07.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.919+0000 7f61577c7700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f61500ce4d0 0x7f61500d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f61577c7700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61500d0150 con 0x7f61500ce910 2024-01-15T11:27:07.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f61577c7700 1 -- --> v1:172.21.15.141:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61500cee50 con 0x7f615015e080 2024-01-15T11:27:07.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f61577c7700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f615015e460 con 0x7f61500ce4d0 2024-01-15T11:27:07.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f6154d62700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f61500ce4d0 0x7f61500d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f6155d64700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f61500ce910 0x7f61500d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:60592/0 (socket says 172.21.15.113:60592) 2024-01-15T11:27:07.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f6155d64700 1 -- 172.21.15.113:0/660319027 learned_addr learned my addr 172.21.15.113:0/660319027 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-15T11:27:07.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f6154d62700 1 -- 172.21.15.113:0/660319027 >> v1:172.21.15.141:6789/0 conn(0x7f615015e080 legacy=0x7f61500cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T11:27:07.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f6154d62700 1 -- 172.21.15.113:0/660319027 >> v1:172.21.15.113:6789/0 conn(0x7f61500ce910 legacy=0x7f61500d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f6154d62700 1 -- 172.21.15.113:0/660319027 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61500cf130 con 0x7f61500ce4d0 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f6147fff700 1 -- 172.21.15.113:0/660319027 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390097251 0 0) 0x7f61500d0150 con 0x7f61500ce910 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f6154d62700 1 --2- 172.21.15.113:0/660319027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61500ce4d0 0x7f61500d1be0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f614c009840 tx=0x7f614c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b2cb07d7962f76 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f6147fff700 1 -- 172.21.15.113:0/660319027 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c004230 con 0x7f61500ce4d0 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f6147fff700 1 -- 172.21.15.113:0/660319027 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614c004390 con 0x7f61500ce4d0 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.920+0000 7f6147fff700 1 -- 172.21.15.113:0/660319027 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c0036a0 con 0x7f61500ce4d0 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.921+0000 7f61577c7700 1 -- 172.21.15.113:0/660319027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61500ce4d0 msgr2=0x7f61500d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.921+0000 7f61577c7700 1 --2- 172.21.15.113:0/660319027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61500ce4d0 0x7f61500d1be0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f614c009840 tx=0x7f614c009b50 comp rx=0 tx=0).stop 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.921+0000 7f61577c7700 1 -- 172.21.15.113:0/660319027 shutdown_connections 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.921+0000 7f61577c7700 1 --2- 172.21.15.113:0/660319027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61500ce4d0 0x7f61500d1be0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.921+0000 7f61577c7700 1 -- 172.21.15.113:0/660319027 >> 172.21.15.113:0/660319027 conn(0x7f6150073130 msgr2=0x7f615009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.921+0000 7f61577c7700 1 -- 172.21.15.113:0/660319027 shutdown_connections 2024-01-15T11:27:07.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.921+0000 7f61577c7700 1 -- 172.21.15.113:0/660319027 wait complete. 2024-01-15T11:27:07.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.922+0000 7f61577c7700 1 Processor -- start 2024-01-15T11:27:07.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.922+0000 7f61577c7700 1 -- start start 2024-01-15T11:27:07.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f61577c7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61500ce4d0 0x7f61500b6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f61577c7700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61500ce910 0x7f61501593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f61577c7700 1 --2- >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f615015e080 0x7f6150159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f61577c7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61500c9a10 con 0x7f61500ce4d0 2024-01-15T11:27:07.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f61577c7700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61500c9cf0 con 0x7f61500ce910 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f61577c7700 1 -- --> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61500c9fd0 con 0x7f615015e080 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6155563700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61500ce4d0 0x7f61500b6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6155563700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61500ce4d0 0x7f61500b6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42174/0 (socket says 172.21.15.113:42174) 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6155563700 1 -- 172.21.15.113:0/3514647339 learned_addr learned my addr 172.21.15.113:0/3514647339 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6155563700 1 -- 172.21.15.113:0/3514647339 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f615015e080 msgr2=0x7f6150159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6155d64700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f615015e080 0x7f6150159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6154d62700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61500ce910 0x7f61501593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6155563700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f615015e080 0x7f6150159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6155563700 1 -- 172.21.15.113:0/3514647339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61500ce910 msgr2=0x7f61501593d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6155563700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61500ce910 0x7f61501593d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6155563700 1 -- 172.21.15.113:0/3514647339 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c0094a0 con 0x7f61500ce4d0 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6155563700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61500ce4d0 0x7f61500b6cb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f614000b4a0 tx=0x7f614000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6145ffb700 1 -- 172.21.15.113:0/3514647339 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140011830 con 0x7f61500ce4d0 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6145ffb700 1 -- 172.21.15.113:0/3514647339 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6140011990 con 0x7f61500ce4d0 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f61577c7700 1 -- 172.21.15.113:0/3514647339 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61500ca2b0 con 0x7f61500ce4d0 2024-01-15T11:27:07.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.923+0000 7f6145ffb700 1 -- 172.21.15.113:0/3514647339 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140011d40 con 0x7f61500ce4d0 2024-01-15T11:27:07.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.924+0000 7f61577c7700 1 -- 172.21.15.113:0/3514647339 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61500caab0 con 0x7f61500ce4d0 2024-01-15T11:27:07.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.925+0000 7f61577c7700 1 -- 172.21.15.113:0/3514647339 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6150067f60 con 0x7f61500ce4d0 2024-01-15T11:27:07.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.925+0000 7f6145ffb700 1 -- 172.21.15.113:0/3514647339 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f614001a400 con 0x7f61500ce4d0 2024-01-15T11:27:07.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.925+0000 7f6145ffb700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f613406b590 0x7f613406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T11:27:07.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.925+0000 7f6154d62700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f613406b590 0x7f613406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T11:27:07.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.925+0000 7f6145ffb700 1 -- 172.21.15.113:0/3514647339 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f6140089980 con 0x7f61500ce4d0 2024-01-15T11:27:07.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.926+0000 7f6154d62700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f613406b590 0x7f613406da50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f61500c5710 tx=0x7f614c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T11:27:07.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:07.930+0000 7f6145ffb700 1 -- 172.21.15.113:0/3514647339 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6140054770 con 0x7f61500ce4d0 2024-01-15T11:27:08.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.135+0000 7f61577c7700 1 -- 172.21.15.113:0/3514647339 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f61500d19c0 con 0x7f61500ce4d0 2024-01-15T11:27:08.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.135+0000 7f6145ffb700 1 -- 172.21.15.113:0/3514647339 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v36) v1 ==== 80+0+122 (secure 0 0 0) 0x7f6140015080 con 0x7f61500ce4d0 2024-01-15T11:27:08.135 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6837/436972884 2024-01-16T11:27:03.723719+0000 2024-01-15T11:27:08.135 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6835/436972884 2024-01-16T11:27:03.723719+0000 2024-01-15T11:27:08.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.137+0000 7f61577c7700 1 -- 172.21.15.113:0/3514647339 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f613406b590 msgr2=0x7f613406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:08.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.137+0000 7f61577c7700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f613406b590 0x7f613406da50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f61500c5710 tx=0x7f614c008030 comp rx=0 tx=0).stop 2024-01-15T11:27:08.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.137+0000 7f61577c7700 1 -- 172.21.15.113:0/3514647339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61500ce4d0 msgr2=0x7f61500b6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T11:27:08.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.137+0000 7f61577c7700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61500ce4d0 0x7f61500b6cb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f614000b4a0 tx=0x7f614000b860 comp rx=0 tx=0).stop 2024-01-15T11:27:08.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.137+0000 7f61577c7700 1 -- 172.21.15.113:0/3514647339 shutdown_connections 2024-01-15T11:27:08.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.137+0000 7f61577c7700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.113:6832/3789331922,v1:172.21.15.113:6833/3789331922] conn(0x7f613406b590 0x7f613406da50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:08.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.137+0000 7f61577c7700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.141:3301/0,v1:172.21.15.141:6790/0] conn(0x7f615015e080 0x7f6150159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:08.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.137+0000 7f61577c7700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61500ce4d0 0x7f61500b6cb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:08.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.137+0000 7f61577c7700 1 --2- 172.21.15.113:0/3514647339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f61500ce910 0x7f61501593d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T11:27:08.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.137+0000 7f61577c7700 1 -- 172.21.15.113:0/3514647339 >> 172.21.15.113:0/3514647339 conn(0x7f6150073130 msgr2=0x7f615006fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T11:27:08.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.137+0000 7f61577c7700 1 -- 172.21.15.113:0/3514647339 shutdown_connections 2024-01-15T11:27:08.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-15T11:27:08.137+0000 7f61577c7700 1 -- 172.21.15.113:0/3514647339 wait complete. 2024-01-15T11:27:08.138 INFO:teuthology.orchestra.run.smithi113.stderr:listed 2 entries 2024-01-15T11:27:08.149 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-15T11:27:08.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T11:27:08.150 DEBUG:teuthology.orchestra.run.smithi113:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-15T11:27:14.344 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:27:14.347 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:27:44.414 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:27:44.418 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:28:14.494 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:28:14.498 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:28:44.575 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:28:44.579 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:29:14.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:29:14.676 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:29:44.759 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:29:44.762 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:30:14.822 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:30:14.827 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:30:44.886 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:30:44.891 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:31:14.955 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:31:14.958 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:31:45.035 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:31:45.039 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:32:08.154 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-01-15T11:32:08.206 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-01-15T11:32:15.121 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:32:15.125 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:32:45.194 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:32:45.198 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:33:15.265 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:33:15.269 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:33:45.327 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:33:45.331 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:34:15.391 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:34:15.395 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:34:45.505 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:34:45.508 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:35:15.575 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:35:15.579 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:35:45.672 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:35:45.676 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:36:15.742 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:36:15.745 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:36:45.817 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:36:45.820 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:37:15.886 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:37:15.891 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:37:45.984 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:37:45.987 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:38:16.081 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:38:16.084 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:38:46.165 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:38:46.168 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:39:16.233 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:39:16.236 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:39:46.340 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:39:46.344 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:40:16.415 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:40:16.418 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:40:46.492 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:40:46.495 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:41:16.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:41:16.566 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:41:46.633 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:41:46.636 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:42:16.713 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:42:16.717 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:42:46.806 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:42:46.810 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:43:16.901 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:43:16.905 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:43:46.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:43:46.982 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:44:17.058 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:44:17.061 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:44:47.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:44:47.156 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:45:17.257 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:45:17.261 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:45:47.344 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:45:47.347 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:46:17.414 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:46:17.417 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:46:47.486 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:46:47.489 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:47:17.582 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:47:17.586 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:47:47.654 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:47:47.658 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:48:17.729 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:48:17.732 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:48:47.797 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:48:47.800 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:49:17.860 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:49:17.864 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:49:47.951 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:49:47.954 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:50:18.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:50:18.064 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:50:48.137 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:50:48.140 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:51:18.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:51:18.213 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:51:48.277 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:51:48.280 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:52:18.364 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:52:18.367 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:52:48.440 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:52:48.444 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:53:18.545 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:53:18.548 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:53:48.616 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:53:48.620 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:54:18.754 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:54:18.757 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:54:48.907 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:54:48.910 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:55:19.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:55:19.007 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:55:49.148 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:55:49.151 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:56:19.295 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:56:19.298 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:56:49.432 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:56:49.435 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:57:19.570 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:57:19.574 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:57:49.690 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:57:49.693 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:58:19.837 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:58:19.841 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:58:49.904 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:58:49.909 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:59:20.079 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:59:20.083 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:59:50.199 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T11:59:50.202 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:00:20.336 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:00:20.339 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:00:50.491 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:00:50.494 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:01:20.637 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:01:20.640 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:01:50.734 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:01:50.738 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:02:20.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:02:20.840 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:02:50.910 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:02:50.914 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:03:21.011 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:03:21.015 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:03:51.166 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:03:51.170 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:04:21.295 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:04:21.298 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:04:51.432 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:04:51.436 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:05:21.603 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:05:21.607 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:05:51.775 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:05:51.780 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:06:21.928 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:06:21.932 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:06:51.991 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:06:51.994 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:07:22.057 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:07:22.060 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:07:52.208 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:07:52.211 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:08:22.286 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:08:22.290 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:08:52.442 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:08:52.445 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:09:22.579 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:09:22.583 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:09:52.741 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:09:52.745 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:10:22.895 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:10:22.898 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:10:53.025 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:10:53.029 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:11:23.171 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:11:23.175 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:11:53.308 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:11:53.312 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:12:23.430 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:12:23.434 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:12:53.584 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:12:53.587 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:13:23.730 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:13:23.734 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:13:53.898 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:13:53.905 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:14:24.042 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:14:24.046 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:14:54.137 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:14:54.142 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:15:24.298 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:15:24.301 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:15:54.419 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:15:54.422 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:16:24.544 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:16:24.547 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:16:54.678 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:16:54.684 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:17:24.832 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:17:24.836 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:17:54.969 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:17:54.972 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:18:25.214 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:18:25.217 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:18:55.333 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:18:55.336 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:19:25.486 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:19:25.490 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:19:55.618 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:19:55.622 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:20:25.789 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:20:25.804 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:20:55.960 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:20:55.964 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:21:26.024 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:21:26.027 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:21:56.185 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:21:56.188 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:22:26.316 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:22:26.320 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:22:56.471 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:22:56.474 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:23:26.626 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:23:26.630 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:23:56.797 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:23:56.800 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:24:26.935 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:24:26.939 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:24:57.077 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:24:57.081 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:25:27.247 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:25:27.251 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:25:57.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:25:57.317 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:26:27.470 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:26:27.474 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:26:57.647 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:26:57.651 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:27:27.785 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:27:27.788 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:27:57.947 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:27:57.950 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:28:28.107 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:28:28.110 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:28:58.248 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:28:58.251 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:29:28.397 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:29:28.400 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:29:58.516 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:29:58.519 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:30:28.678 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:30:28.681 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:30:58.839 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:30:58.842 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:31:28.983 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:31:28.986 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:31:59.149 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:31:59.153 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:32:29.307 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:32:29.310 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:32:59.448 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:32:59.454 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:33:29.596 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:33:29.601 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:33:59.752 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:33:59.755 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:34:29.886 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:34:29.889 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:35:00.041 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:35:00.045 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:35:30.120 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:35:30.123 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:36:00.233 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:36:00.236 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:36:30.396 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:36:30.399 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:37:00.541 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:37:00.545 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:37:30.672 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:37:30.676 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:38:00.817 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:38:00.820 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:38:30.946 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:38:30.949 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:39:01.089 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:39:01.092 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:39:31.226 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:39:31.230 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:40:01.350 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:40:01.353 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:40:31.506 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:40:31.509 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:41:01.642 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:41:01.645 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:41:31.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:41:31.789 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:42:01.943 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:42:01.946 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:42:32.088 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:42:32.091 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:43:02.247 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:43:02.250 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:43:32.315 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:43:32.318 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:44:02.468 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:44:02.471 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:44:32.606 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:44:32.609 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:45:02.777 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:45:02.781 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:45:32.936 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:45:32.940 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:46:03.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:46:03.063 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:46:33.205 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:46:33.209 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:47:03.359 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:47:03.363 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:47:33.448 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:47:33.452 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:48:03.642 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:48:03.645 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:48:33.765 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:48:33.769 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:49:03.911 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:49:03.916 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:49:34.065 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:49:34.068 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:50:04.205 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:50:04.208 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:50:34.367 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:50:34.370 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:51:04.511 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:51:04.514 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:51:34.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:51:34.669 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:52:04.813 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:52:04.816 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:52:34.942 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:52:34.945 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:53:05.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:53:05.101 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:53:35.219 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:53:35.222 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:54:05.363 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:54:05.367 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:54:35.502 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:54:35.505 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:55:05.655 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:55:05.659 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:55:35.802 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:55:35.805 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:56:05.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:56:05.937 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:56:36.086 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:56:36.089 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:57:06.307 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:57:06.310 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:57:36.419 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:57:36.424 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:58:06.561 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:58:06.565 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:58:36.712 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:58:36.716 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:59:06.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:59:06.879 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:59:37.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T12:59:37.033 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:00:07.151 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:00:07.154 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:00:37.239 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:00:37.242 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:01:07.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:01:07.409 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:01:37.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:01:37.554 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:02:07.662 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:02:07.665 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:02:37.812 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:02:37.817 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:03:07.962 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:03:07.965 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:03:38.051 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:03:38.054 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:04:08.123 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:04:08.127 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:04:38.259 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:04:38.262 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:05:08.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:05:08.407 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:05:38.576 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:05:38.580 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:06:08.722 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:06:08.726 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:06:38.880 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:06:38.884 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:07:09.023 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:07:09.027 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:07:39.178 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:07:39.181 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:08:09.305 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:08:09.308 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:08:39.443 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:08:39.447 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:09:09.590 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:09:09.593 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:09:39.728 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:09:39.732 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:10:09.882 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:10:09.885 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:10:40.050 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:10:40.054 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:11:10.219 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:11:10.224 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:11:40.339 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:11:40.342 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:12:10.483 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:12:10.487 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:12:40.637 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:12:40.641 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:13:10.799 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:13:10.802 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:13:41.021 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:13:41.024 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:14:11.150 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:14:11.154 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:14:41.255 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:14:41.258 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:15:11.408 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:15:11.412 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:15:41.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:15:41.566 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:16:11.667 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:16:11.670 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:16:41.788 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:16:41.791 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:17:11.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:17:11.937 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:17:42.076 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:17:42.080 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:18:12.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:18:12.213 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:18:42.358 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:18:42.362 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:19:12.527 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:19:12.530 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:19:42.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:19:42.669 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:20:12.779 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:20:12.783 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:20:42.858 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:20:42.861 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:21:12.992 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:21:12.996 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:21:43.112 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:21:43.115 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:22:13.248 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:22:13.251 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:22:43.389 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:22:43.399 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:23:13.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:23:13.559 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:23:43.676 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:23:43.681 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:24:13.819 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:24:13.823 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:24:43.914 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:24:43.919 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:25:14.045 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:25:14.050 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:25:44.200 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:25:44.204 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:26:14.338 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:26:14.342 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:26:44.467 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:26:44.470 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:27:14.622 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:27:14.625 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:27:44.735 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:27:44.738 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:28:14.815 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:28:14.819 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:28:44.929 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:28:44.934 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:29:15.067 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:29:15.071 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:29:45.221 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:29:45.224 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:30:15.324 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:30:15.328 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:30:45.478 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:30:45.482 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:31:15.617 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:31:15.621 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:31:45.761 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:31:45.765 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:32:15.908 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:32:15.916 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:32:45.988 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:32:45.997 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:33:16.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:33:16.159 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:33:46.296 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:33:46.299 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:34:16.431 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:34:16.440 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:34:46.575 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:34:46.579 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:35:16.728 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:35:16.731 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:35:46.899 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:35:46.903 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:36:17.019 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:36:17.022 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:36:47.159 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:36:47.162 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:37:17.302 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:37:17.305 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:37:47.423 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:37:47.427 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:38:17.556 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:38:17.559 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:38:47.709 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:38:47.713 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:39:17.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:39:17.879 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:39:48.019 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:39:48.024 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:40:18.165 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:40:18.169 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:40:48.323 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:40:48.327 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:41:18.461 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:41:18.467 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:41:48.606 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:41:48.611 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:42:18.751 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:42:18.755 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:42:48.879 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:42:48.882 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:43:19.019 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:43:19.022 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:43:49.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:43:49.161 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:44:19.297 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:44:19.301 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:44:49.451 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:44:49.455 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:45:19.589 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:45:19.592 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:45:49.768 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:45:49.772 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:46:19.924 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:46:19.928 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:46:50.070 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:46:50.073 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:47:20.201 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:47:20.205 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:47:50.355 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:47:50.358 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:48:20.482 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:48:20.489 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:48:50.571 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:48:50.574 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:49:20.733 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:49:20.737 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:49:50.942 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:49:50.946 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:50:21.095 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:50:21.099 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:50:51.190 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:50:51.193 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:51:21.334 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:51:21.338 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:51:51.454 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:51:51.458 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:52:21.775 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:52:21.778 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:52:51.923 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:52:51.926 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:53:22.069 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:53:22.071 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:53:52.215 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:53:52.218 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:54:22.280 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:54:22.290 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:54:52.449 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:54:52.452 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:55:22.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:55:22.598 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:55:52.950 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:55:52.954 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:56:23.219 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:56:23.223 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:56:53.284 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:56:53.288 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:57:23.436 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:57:23.443 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:57:53.586 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:57:53.591 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:58:23.651 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:58:23.657 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:58:53.960 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:58:53.965 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:59:24.110 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:59:24.115 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:59:54.247 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T13:59:54.250 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:00:24.302 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:00:24.306 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:00:54.389 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:00:54.393 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:01:24.545 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:01:24.548 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:01:54.693 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:01:54.700 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:02:24.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:02:24.871 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:02:55.003 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:02:55.007 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:03:25.147 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:03:25.150 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:03:55.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:03:55.278 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:04:25.396 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:04:25.401 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:04:55.499 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:04:55.507 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:05:25.643 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:05:25.646 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:05:55.779 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:05:55.783 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:06:25.932 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:06:25.936 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:06:56.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:06:56.063 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:07:26.211 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:07:26.216 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:07:56.366 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:07:56.370 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:08:26.519 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:08:26.522 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:08:56.715 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:08:56.721 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:09:26.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:09:26.880 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:09:57.032 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:09:57.036 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:10:27.162 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:10:27.167 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:10:57.259 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:10:57.262 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:11:27.363 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:11:27.366 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:11:57.501 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:11:57.504 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:12:27.631 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:12:27.634 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:12:57.766 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:12:57.771 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:13:27.914 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:13:27.920 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:13:58.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:13:58.064 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:14:28.235 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:14:28.241 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:14:58.387 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:14:58.391 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:15:28.514 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:15:28.519 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:15:58.717 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:15:58.721 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:16:28.874 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:16:28.877 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:16:59.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:16:59.034 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:17:29.180 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:17:29.183 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:17:59.337 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:17:59.341 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:18:29.458 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:18:29.461 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:18:59.597 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:18:59.601 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:19:29.947 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:19:29.950 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:20:00.081 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:20:00.085 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:20:30.236 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:20:30.240 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:21:00.391 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:21:00.395 DEBUG:teuthology.orchestra.run.smithi141:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:21:09.877 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:21:09.879 DEBUG:tasks.cephfs.kernel_mount:Killing processes on client.0... 2024-01-15T14:21:09.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:21:09.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:21:09.883 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:21:13.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-15T14:21:13.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-01-15T14:21:13.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-01-15T14:21:13.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-01-15T14:21:13.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-01-15T14:21:13.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-01-15T14:21:13.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-01-15T14:21:13.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-01-15T14:21:13.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-01-15T14:21:13.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-01-15T14:21:13.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-01-15T14:21:13.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-01-15T14:21:13.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-01-15T14:21:13.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-01-15T14:21:13.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-01-15T14:21:13.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-01-15T14:21:13.868 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-01-15T14:21:18.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-01-15T14:21:18.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-01-15T14:21:18.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2024-01-15T14:21:18.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~5s 2024-01-15T14:21:18.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~5s 2024-01-15T14:21:18.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~5s 2024-01-15T14:21:18.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~5s 2024-01-15T14:21:18.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~5s 2024-01-15T14:21:18.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~5s 2024-01-15T14:21:18.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~5s 2024-01-15T14:21:18.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~5s 2024-01-15T14:21:18.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~5s 2024-01-15T14:21:18.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~5s 2024-01-15T14:21:18.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~5s 2024-01-15T14:21:18.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~5s 2024-01-15T14:21:18.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~5s 2024-01-15T14:21:18.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~5s 2024-01-15T14:21:18.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:21:18.884 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:21:23.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~10s 2024-01-15T14:21:23.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~10s 2024-01-15T14:21:28.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~15s 2024-01-15T14:21:28.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~15s 2024-01-15T14:21:28.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~15s 2024-01-15T14:21:28.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~15s 2024-01-15T14:21:30.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:21:30.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:21:33.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~20s 2024-01-15T14:21:33.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~20s 2024-01-15T14:21:33.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~20s 2024-01-15T14:21:33.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~20s 2024-01-15T14:21:33.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~20s 2024-01-15T14:21:33.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~20s 2024-01-15T14:21:33.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~20s 2024-01-15T14:21:33.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~20s 2024-01-15T14:21:33.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~20s 2024-01-15T14:21:33.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~20s 2024-01-15T14:21:33.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~20s 2024-01-15T14:21:33.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~20s 2024-01-15T14:21:33.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~20s 2024-01-15T14:21:33.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~20s 2024-01-15T14:21:33.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~20s 2024-01-15T14:21:33.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~20s 2024-01-15T14:21:33.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~20s 2024-01-15T14:21:38.896 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~25s 2024-01-15T14:21:38.896 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~25s 2024-01-15T14:21:38.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~25s 2024-01-15T14:21:43.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~30s 2024-01-15T14:21:43.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~30s 2024-01-15T14:21:43.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~30s 2024-01-15T14:21:43.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~30s 2024-01-15T14:21:43.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~30s 2024-01-15T14:21:43.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~30s 2024-01-15T14:21:43.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~30s 2024-01-15T14:21:43.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~30s 2024-01-15T14:21:43.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~30s 2024-01-15T14:21:43.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~30s 2024-01-15T14:21:43.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~30s 2024-01-15T14:21:43.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~30s 2024-01-15T14:21:43.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~30s 2024-01-15T14:21:43.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~30s 2024-01-15T14:21:43.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~30s 2024-01-15T14:21:43.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~30s 2024-01-15T14:21:43.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~30s 2024-01-15T14:21:48.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~35s 2024-01-15T14:21:48.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~35s 2024-01-15T14:21:48.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~35s 2024-01-15T14:21:48.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~35s 2024-01-15T14:21:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~35s 2024-01-15T14:21:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~35s 2024-01-15T14:21:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~35s 2024-01-15T14:21:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~35s 2024-01-15T14:21:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~35s 2024-01-15T14:21:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~35s 2024-01-15T14:21:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~35s 2024-01-15T14:21:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~35s 2024-01-15T14:21:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~35s 2024-01-15T14:21:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~35s 2024-01-15T14:21:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~35s 2024-01-15T14:21:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~35s 2024-01-15T14:21:48.910 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~35s 2024-01-15T14:21:48.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:21:53.340 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:21:53.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~40s 2024-01-15T14:21:53.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~40s 2024-01-15T14:21:53.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~40s 2024-01-15T14:21:53.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~40s 2024-01-15T14:21:53.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~40s 2024-01-15T14:21:57.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:21:57.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:21:58.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~45s 2024-01-15T14:21:58.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~45s 2024-01-15T14:21:58.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~45s 2024-01-15T14:21:58.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~45s 2024-01-15T14:21:58.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~45s 2024-01-15T14:21:58.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~45s 2024-01-15T14:21:58.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~45s 2024-01-15T14:21:58.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~45s 2024-01-15T14:21:58.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~45s 2024-01-15T14:21:58.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~45s 2024-01-15T14:21:58.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~45s 2024-01-15T14:21:58.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~45s 2024-01-15T14:21:58.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~45s 2024-01-15T14:21:58.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~45s 2024-01-15T14:21:58.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~45s 2024-01-15T14:21:58.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~45s 2024-01-15T14:21:58.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~45s 2024-01-15T14:22:01.052 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:22:03.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~50s 2024-01-15T14:22:03.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~50s 2024-01-15T14:22:03.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~50s 2024-01-15T14:22:03.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~50s 2024-01-15T14:22:03.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~50s 2024-01-15T14:22:03.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~50s 2024-01-15T14:22:03.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~50s 2024-01-15T14:22:03.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~50s 2024-01-15T14:22:03.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~50s 2024-01-15T14:22:03.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~50s 2024-01-15T14:22:03.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~50s 2024-01-15T14:22:03.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~50s 2024-01-15T14:22:03.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~50s 2024-01-15T14:22:03.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~50s 2024-01-15T14:22:03.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~50s 2024-01-15T14:22:03.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~50s 2024-01-15T14:22:03.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~50s 2024-01-15T14:22:05.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:22:05.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:22:08.412 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~55s 2024-01-15T14:22:08.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~55s 2024-01-15T14:22:08.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~55s 2024-01-15T14:22:08.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~55s 2024-01-15T14:22:08.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~55s 2024-01-15T14:22:11.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:22:11.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:22:11.484 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:22:11.484 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:22:11.488 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed ... ERROR 2024-01-15T14:22:11.488 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.488 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-15T14:22:11.488 INFO:tasks.cephfs_test_runner:ERROR: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T14:22:11.488 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed 2024-01-15T14:22:11.488 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T14:22:11.488 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T14:22:11.488 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 160, in umount 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 148, in wait 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get 2024-01-15T14:22:11.489 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 323, in copy_file_to 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 276, in copy_to_log 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: for line in f: 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: line = self.readline() 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner:socket.timeout 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.490 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7609, in tearDown 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: super(self.__class__, self).tearDown() 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 459, in tearDown 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: super(TestVolumesHelper, self).tearDown() 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/cephfs_test_case.py", line 205, in tearDown 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: m.teardown() 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 207, in teardown 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: self.umount() 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 163, in umount 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: self.client_remote.run( 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner:ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T14:22:11.491 INFO:tasks.cephfs_test_runner:Ran 1 test in 10585.919s 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner:ERROR: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2024-01-15T14:22:11.492 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 160, in umount 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 148, in wait 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 323, in copy_file_to 2024-01-15T14:22:11.493 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 276, in copy_to_log 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: for line in f: 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: line = self.readline() 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner:socket.timeout 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7609, in tearDown 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: super(self.__class__, self).tearDown() 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 459, in tearDown 2024-01-15T14:22:11.494 INFO:tasks.cephfs_test_runner: super(TestVolumesHelper, self).tearDown() 2024-01-15T14:22:11.495 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/cephfs_test_case.py", line 205, in tearDown 2024-01-15T14:22:11.495 INFO:tasks.cephfs_test_runner: m.teardown() 2024-01-15T14:22:11.495 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 207, in teardown 2024-01-15T14:22:11.495 INFO:tasks.cephfs_test_runner: self.umount() 2024-01-15T14:22:11.495 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 163, in umount 2024-01-15T14:22:11.495 INFO:tasks.cephfs_test_runner: self.client_remote.run( 2024-01-15T14:22:11.495 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run 2024-01-15T14:22:11.495 INFO:tasks.cephfs_test_runner: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-01-15T14:22:11.495 INFO:tasks.cephfs_test_runner:ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:22:11.495 INFO:tasks.cephfs_test_runner: 2024-01-15T14:22:11.495 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T14:22:11.722 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=03572511b5684e238e975862f0858046 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T14:22:11.723 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-01-15T14:22:11.732 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-01-15T14:22:11.740 INFO:tasks.kclient:Unmounting kernel clients... 2024-01-15T14:22:11.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:22:11.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:22:13.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:22:13.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:22:13.927 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~60s 2024-01-15T14:22:13.927 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~60s 2024-01-15T14:22:13.927 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~60s 2024-01-15T14:22:13.927 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~60s 2024-01-15T14:22:13.927 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~60s 2024-01-15T14:22:13.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~60s 2024-01-15T14:22:13.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~60s 2024-01-15T14:22:13.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~60s 2024-01-15T14:22:13.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~60s 2024-01-15T14:22:13.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~60s 2024-01-15T14:22:13.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~60s 2024-01-15T14:22:13.929 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~60s 2024-01-15T14:22:13.929 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~60s 2024-01-15T14:22:13.929 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~60s 2024-01-15T14:22:13.929 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~60s 2024-01-15T14:22:13.929 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~60s 2024-01-15T14:22:13.929 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~60s 2024-01-15T14:22:14.812 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:22:14.813 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:22:17.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:22:17.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:22:17.884 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:22:17.884 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:22:17.885 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 184, in invoke_logrotate procs = ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:22:18.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~65s 2024-01-15T14:22:18.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s 2024-01-15T14:22:18.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~65s 2024-01-15T14:22:18.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~65s 2024-01-15T14:22:18.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~65s 2024-01-15T14:22:18.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~65s 2024-01-15T14:22:18.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~65s 2024-01-15T14:22:18.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~65s 2024-01-15T14:22:18.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~65s 2024-01-15T14:22:18.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~65s 2024-01-15T14:22:18.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~65s 2024-01-15T14:22:18.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~65s 2024-01-15T14:22:18.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~65s 2024-01-15T14:22:18.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~65s 2024-01-15T14:22:18.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~65s 2024-01-15T14:22:18.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~65s 2024-01-15T14:22:18.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~65s 2024-01-15T14:22:23.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:22:23.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:22:23.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~70s 2024-01-15T14:22:23.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~70s 2024-01-15T14:22:23.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~70s 2024-01-15T14:22:23.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~70s 2024-01-15T14:22:23.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~70s 2024-01-15T14:22:23.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~70s 2024-01-15T14:22:23.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~70s 2024-01-15T14:22:23.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~70s 2024-01-15T14:22:23.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~70s 2024-01-15T14:22:23.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~70s 2024-01-15T14:22:23.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~70s 2024-01-15T14:22:23.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~70s 2024-01-15T14:22:23.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~70s 2024-01-15T14:22:23.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~70s 2024-01-15T14:22:23.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~70s 2024-01-15T14:22:23.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~70s 2024-01-15T14:22:23.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~70s 2024-01-15T14:22:26.876 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:22:28.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~75s 2024-01-15T14:22:28.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~75s 2024-01-15T14:22:28.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~75s 2024-01-15T14:22:28.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~75s 2024-01-15T14:22:28.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~75s 2024-01-15T14:22:28.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~75s 2024-01-15T14:22:28.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~75s 2024-01-15T14:22:28.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~75s 2024-01-15T14:22:28.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~75s 2024-01-15T14:22:28.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~75s 2024-01-15T14:22:28.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~75s 2024-01-15T14:22:28.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~75s 2024-01-15T14:22:28.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~75s 2024-01-15T14:22:28.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~75s 2024-01-15T14:22:28.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~75s 2024-01-15T14:22:28.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~75s 2024-01-15T14:22:28.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~75s 2024-01-15T14:22:33.952 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2024-01-15T14:22:33.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~80s 2024-01-15T14:22:33.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~80s 2024-01-15T14:22:33.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~80s 2024-01-15T14:22:33.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~80s 2024-01-15T14:22:33.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~80s 2024-01-15T14:22:33.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~80s 2024-01-15T14:22:33.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~80s 2024-01-15T14:22:33.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~80s 2024-01-15T14:22:33.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~80s 2024-01-15T14:22:33.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~80s 2024-01-15T14:22:33.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~80s 2024-01-15T14:22:33.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~80s 2024-01-15T14:22:33.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~80s 2024-01-15T14:22:33.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~80s 2024-01-15T14:22:33.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~80s 2024-01-15T14:22:33.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~80s 2024-01-15T14:22:38.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:22:38.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:22:38.958 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2024-01-15T14:22:38.958 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~85s 2024-01-15T14:22:38.958 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~85s 2024-01-15T14:22:38.958 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~85s 2024-01-15T14:22:38.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~85s 2024-01-15T14:22:38.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~85s 2024-01-15T14:22:38.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~85s 2024-01-15T14:22:38.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~85s 2024-01-15T14:22:38.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~85s 2024-01-15T14:22:38.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~85s 2024-01-15T14:22:38.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~85s 2024-01-15T14:22:38.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~85s 2024-01-15T14:22:38.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~85s 2024-01-15T14:22:38.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~85s 2024-01-15T14:22:38.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~85s 2024-01-15T14:22:38.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~85s 2024-01-15T14:22:38.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~85s 2024-01-15T14:22:41.948 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:22:43.963 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~90s 2024-01-15T14:22:43.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~90s 2024-01-15T14:22:43.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~90s 2024-01-15T14:22:43.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~90s 2024-01-15T14:22:43.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~90s 2024-01-15T14:22:43.965 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~90s 2024-01-15T14:22:43.965 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~90s 2024-01-15T14:22:43.965 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~90s 2024-01-15T14:22:43.965 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~90s 2024-01-15T14:22:43.965 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~90s 2024-01-15T14:22:43.965 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~90s 2024-01-15T14:22:43.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~90s 2024-01-15T14:22:43.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~90s 2024-01-15T14:22:43.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~90s 2024-01-15T14:22:43.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~90s 2024-01-15T14:22:43.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~90s 2024-01-15T14:22:43.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~90s 2024-01-15T14:22:44.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:22:44.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:22:45.020 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:22:45.020 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:22:45.021 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:22:45.028 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-15T14:22:45.037 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-15T14:22:45.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:22:45.038 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:22:48.092 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:22:48.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~95s 2024-01-15T14:22:48.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~95s 2024-01-15T14:22:48.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~95s 2024-01-15T14:22:48.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~95s 2024-01-15T14:22:48.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~95s 2024-01-15T14:22:48.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~95s 2024-01-15T14:22:48.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~95s 2024-01-15T14:22:48.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~95s 2024-01-15T14:22:48.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~95s 2024-01-15T14:22:48.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~95s 2024-01-15T14:22:48.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~95s 2024-01-15T14:22:48.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~95s 2024-01-15T14:22:48.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~95s 2024-01-15T14:22:48.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~95s 2024-01-15T14:22:48.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~95s 2024-01-15T14:22:48.972 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~95s 2024-01-15T14:22:48.972 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~95s 2024-01-15T14:22:53.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~100s 2024-01-15T14:22:53.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~100s 2024-01-15T14:22:53.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~100s 2024-01-15T14:22:53.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~100s 2024-01-15T14:22:53.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~100s 2024-01-15T14:22:53.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~100s 2024-01-15T14:22:53.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~100s 2024-01-15T14:22:53.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~100s 2024-01-15T14:22:53.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~100s 2024-01-15T14:22:53.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~100s 2024-01-15T14:22:53.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~100s 2024-01-15T14:22:53.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~100s 2024-01-15T14:22:53.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~100s 2024-01-15T14:22:53.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~100s 2024-01-15T14:22:53.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~100s 2024-01-15T14:22:53.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~100s 2024-01-15T14:22:53.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~100s 2024-01-15T14:22:57.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:22:57.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:22:58.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~105s 2024-01-15T14:22:58.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~105s 2024-01-15T14:22:58.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~105s 2024-01-15T14:22:58.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~105s 2024-01-15T14:22:58.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~105s 2024-01-15T14:22:58.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~105s 2024-01-15T14:22:58.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~105s 2024-01-15T14:22:58.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~105s 2024-01-15T14:22:58.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~105s 2024-01-15T14:22:58.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~105s 2024-01-15T14:22:58.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~105s 2024-01-15T14:22:58.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~105s 2024-01-15T14:22:58.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~105s 2024-01-15T14:22:58.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~105s 2024-01-15T14:22:58.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~105s 2024-01-15T14:22:58.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~105s 2024-01-15T14:22:58.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~105s 2024-01-15T14:23:00.156 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:23:03.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~110s 2024-01-15T14:23:03.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~110s 2024-01-15T14:23:03.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~110s 2024-01-15T14:23:03.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~110s 2024-01-15T14:23:03.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~110s 2024-01-15T14:23:03.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~110s 2024-01-15T14:23:03.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~110s 2024-01-15T14:23:03.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~110s 2024-01-15T14:23:03.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~110s 2024-01-15T14:23:03.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~110s 2024-01-15T14:23:03.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~110s 2024-01-15T14:23:03.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~110s 2024-01-15T14:23:03.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~110s 2024-01-15T14:23:03.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~110s 2024-01-15T14:23:03.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~110s 2024-01-15T14:23:03.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~110s 2024-01-15T14:23:03.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~110s 2024-01-15T14:23:09.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~115s 2024-01-15T14:23:09.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~115s 2024-01-15T14:23:09.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~115s 2024-01-15T14:23:09.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~115s 2024-01-15T14:23:09.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~115s 2024-01-15T14:23:09.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~115s 2024-01-15T14:23:09.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~115s 2024-01-15T14:23:09.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~115s 2024-01-15T14:23:09.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~115s 2024-01-15T14:23:09.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~115s 2024-01-15T14:23:09.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~115s 2024-01-15T14:23:09.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~115s 2024-01-15T14:23:09.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~115s 2024-01-15T14:23:09.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~115s 2024-01-15T14:23:09.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~115s 2024-01-15T14:23:09.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~115s 2024-01-15T14:23:09.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~115s 2024-01-15T14:23:12.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:23:12.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:23:14.009 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~120s 2024-01-15T14:23:14.009 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~120s 2024-01-15T14:23:14.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~120s 2024-01-15T14:23:14.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~120s 2024-01-15T14:23:14.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~120s 2024-01-15T14:23:14.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~120s 2024-01-15T14:23:14.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~120s 2024-01-15T14:23:14.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~120s 2024-01-15T14:23:14.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~120s 2024-01-15T14:23:14.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~120s 2024-01-15T14:23:14.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~120s 2024-01-15T14:23:14.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~120s 2024-01-15T14:23:14.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~120s 2024-01-15T14:23:14.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~120s 2024-01-15T14:23:14.012 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~120s 2024-01-15T14:23:14.012 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~120s 2024-01-15T14:23:14.012 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~120s 2024-01-15T14:23:15.228 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:23:18.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:23:18.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:23:18.300 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:23:18.300 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:23:18.301 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:23:18.316 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-15T14:23:18.316 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T14:23:18.316 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.319 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.320 ERROR:teuthology.misc:Saw exception from ceph.mds.a Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-01-15T14:23:18.338 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T14:23:18.338 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.338 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.339 ERROR:teuthology.misc:Saw exception from ceph.mds.c Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c' 2024-01-15T14:23:18.352 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T14:23:18.352 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.353 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.353 ERROR:teuthology.misc:Saw exception from ceph.mds.b Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b' 2024-01-15T14:23:18.366 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T14:23:18.366 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.366 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.366 ERROR:teuthology.misc:Saw exception from ceph.mds.d Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' 2024-01-15T14:23:18.379 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-15T14:23:18.379 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-15T14:23:18.379 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.380 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.380 ERROR:teuthology.misc:Saw exception from ceph.osd.0 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' 2024-01-15T14:23:18.396 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-15T14:23:18.396 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.396 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.397 ERROR:teuthology.misc:Saw exception from ceph.osd.1 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1' 2024-01-15T14:23:18.413 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-15T14:23:18.413 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.413 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.413 ERROR:teuthology.misc:Saw exception from ceph.osd.2 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2' 2024-01-15T14:23:18.430 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-15T14:23:18.430 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.430 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.430 ERROR:teuthology.misc:Saw exception from ceph.osd.3 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3' 2024-01-15T14:23:18.447 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-15T14:23:18.447 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.447 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.447 ERROR:teuthology.misc:Saw exception from ceph.osd.4 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4' 2024-01-15T14:23:18.463 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-15T14:23:18.463 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.463 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.463 ERROR:teuthology.misc:Saw exception from ceph.osd.5 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5' 2024-01-15T14:23:18.481 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-15T14:23:18.481 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.481 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.481 ERROR:teuthology.misc:Saw exception from ceph.osd.6 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6' 2024-01-15T14:23:18.498 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-15T14:23:18.498 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.498 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.498 ERROR:teuthology.misc:Saw exception from ceph.osd.7 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' 2024-01-15T14:23:18.515 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-15T14:23:18.515 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-15T14:23:18.515 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.515 ERROR:teuthology.misc:Saw exception from ceph.mgr.y Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2024-01-15T14:23:18.535 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-15T14:23:18.535 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.536 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.536 ERROR:teuthology.misc:Saw exception from ceph.mgr.x Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2024-01-15T14:23:18.556 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-15T14:23:18.556 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-15T14:23:18.556 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.556 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.556 ERROR:teuthology.misc:Saw exception from ceph.mon.a Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a' 2024-01-15T14:23:18.580 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-15T14:23:18.581 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.581 ERROR:teuthology.misc:Saw exception from ceph.mon.b Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b' 2024-01-15T14:23:18.606 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-15T14:23:18.606 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:23:18.606 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T14:23:18.606 ERROR:teuthology.misc:Saw exception from ceph.mon.c Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi141 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c' 2024-01-15T14:23:18.632 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-15T14:23:18.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:23:18.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:23:19.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~125s 2024-01-15T14:23:19.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~125s 2024-01-15T14:23:19.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~125s 2024-01-15T14:23:19.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~125s 2024-01-15T14:23:19.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~125s 2024-01-15T14:23:19.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~125s 2024-01-15T14:23:19.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~125s 2024-01-15T14:23:19.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~125s 2024-01-15T14:23:19.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~125s 2024-01-15T14:23:19.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~125s 2024-01-15T14:23:19.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~125s 2024-01-15T14:23:19.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~125s 2024-01-15T14:23:19.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~125s 2024-01-15T14:23:19.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~125s 2024-01-15T14:23:19.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~125s 2024-01-15T14:23:19.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~125s 2024-01-15T14:23:19.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~125s 2024-01-15T14:23:21.692 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:23:24.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~130s 2024-01-15T14:23:24.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~130s 2024-01-15T14:23:24.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~130s 2024-01-15T14:23:24.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~130s 2024-01-15T14:23:24.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~130s 2024-01-15T14:23:24.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~130s 2024-01-15T14:23:24.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~130s 2024-01-15T14:23:24.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~130s 2024-01-15T14:23:24.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~130s 2024-01-15T14:23:24.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~130s 2024-01-15T14:23:24.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~130s 2024-01-15T14:23:24.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~130s 2024-01-15T14:23:24.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~130s 2024-01-15T14:23:24.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~130s 2024-01-15T14:23:24.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~130s 2024-01-15T14:23:24.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~130s 2024-01-15T14:23:24.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~130s 2024-01-15T14:23:29.025 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~135s 2024-01-15T14:23:29.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~135s 2024-01-15T14:23:29.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~135s 2024-01-15T14:23:29.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~135s 2024-01-15T14:23:29.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~135s 2024-01-15T14:23:29.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~135s 2024-01-15T14:23:29.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~135s 2024-01-15T14:23:29.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~135s 2024-01-15T14:23:29.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~135s 2024-01-15T14:23:29.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~135s 2024-01-15T14:23:29.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~135s 2024-01-15T14:23:29.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~135s 2024-01-15T14:23:29.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~135s 2024-01-15T14:23:29.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~135s 2024-01-15T14:23:29.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~135s 2024-01-15T14:23:29.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~135s 2024-01-15T14:23:29.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~135s 2024-01-15T14:23:30.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:23:30.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:23:33.756 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:23:34.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~140s 2024-01-15T14:23:34.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~140s 2024-01-15T14:23:34.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~140s 2024-01-15T14:23:34.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~140s 2024-01-15T14:23:34.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~140s 2024-01-15T14:23:34.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~140s 2024-01-15T14:23:34.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~140s 2024-01-15T14:23:34.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~140s 2024-01-15T14:23:34.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~140s 2024-01-15T14:23:34.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~140s 2024-01-15T14:23:34.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~140s 2024-01-15T14:23:34.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~140s 2024-01-15T14:23:34.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~140s 2024-01-15T14:23:34.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~140s 2024-01-15T14:23:34.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~140s 2024-01-15T14:23:34.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~140s 2024-01-15T14:23:34.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~140s 2024-01-15T14:23:39.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~145s 2024-01-15T14:23:39.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~145s 2024-01-15T14:23:39.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~145s 2024-01-15T14:23:39.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~145s 2024-01-15T14:23:39.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~145s 2024-01-15T14:23:39.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~145s 2024-01-15T14:23:39.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~145s 2024-01-15T14:23:39.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~145s 2024-01-15T14:23:39.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~145s 2024-01-15T14:23:39.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~145s 2024-01-15T14:23:39.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~145s 2024-01-15T14:23:39.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~145s 2024-01-15T14:23:39.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~145s 2024-01-15T14:23:39.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~145s 2024-01-15T14:23:39.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~145s 2024-01-15T14:23:39.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~145s 2024-01-15T14:23:39.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~145s 2024-01-15T14:23:44.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~150s 2024-01-15T14:23:44.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~150s 2024-01-15T14:23:44.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~150s 2024-01-15T14:23:44.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~150s 2024-01-15T14:23:44.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~150s 2024-01-15T14:23:44.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~150s 2024-01-15T14:23:44.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~150s 2024-01-15T14:23:44.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~150s 2024-01-15T14:23:44.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~150s 2024-01-15T14:23:44.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~150s 2024-01-15T14:23:44.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~150s 2024-01-15T14:23:44.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~150s 2024-01-15T14:23:44.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~150s 2024-01-15T14:23:44.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~150s 2024-01-15T14:23:44.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~150s 2024-01-15T14:23:44.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~150s 2024-01-15T14:23:44.049 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~150s 2024-01-15T14:23:45.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:23:45.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:23:48.828 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:23:49.050 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~155s 2024-01-15T14:23:49.050 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~155s 2024-01-15T14:23:49.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~155s 2024-01-15T14:23:49.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~155s 2024-01-15T14:23:49.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~155s 2024-01-15T14:23:49.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~155s 2024-01-15T14:23:49.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~155s 2024-01-15T14:23:49.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~155s 2024-01-15T14:23:49.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~155s 2024-01-15T14:23:49.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~155s 2024-01-15T14:23:49.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~155s 2024-01-15T14:23:49.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~155s 2024-01-15T14:23:49.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~155s 2024-01-15T14:23:49.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~155s 2024-01-15T14:23:49.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~155s 2024-01-15T14:23:49.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~155s 2024-01-15T14:23:49.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~155s 2024-01-15T14:23:51.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:23:51.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:23:51.900 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:23:51.900 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:23:51.900 INFO:tasks.ceph:Archiving crash dumps... 2024-01-15T14:23:51.902 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517064/remote/smithi113/crash 2024-01-15T14:23:51.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:23:51.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:23:54.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~160s 2024-01-15T14:23:54.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~160s 2024-01-15T14:23:54.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~160s 2024-01-15T14:23:54.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~160s 2024-01-15T14:23:54.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~160s 2024-01-15T14:23:54.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~160s 2024-01-15T14:23:54.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~160s 2024-01-15T14:23:54.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~160s 2024-01-15T14:23:54.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~160s 2024-01-15T14:23:54.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~160s 2024-01-15T14:23:54.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~160s 2024-01-15T14:23:54.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~160s 2024-01-15T14:23:54.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~160s 2024-01-15T14:23:54.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~160s 2024-01-15T14:23:54.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~160s 2024-01-15T14:23:54.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~160s 2024-01-15T14:23:54.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~160s 2024-01-15T14:23:54.972 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:23:59.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~165s 2024-01-15T14:23:59.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~165s 2024-01-15T14:23:59.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~165s 2024-01-15T14:23:59.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~165s 2024-01-15T14:23:59.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~165s 2024-01-15T14:23:59.063 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~165s 2024-01-15T14:23:59.063 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~165s 2024-01-15T14:23:59.063 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~165s 2024-01-15T14:23:59.063 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~165s 2024-01-15T14:23:59.063 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~165s 2024-01-15T14:23:59.063 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~165s 2024-01-15T14:23:59.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~165s 2024-01-15T14:23:59.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~165s 2024-01-15T14:23:59.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~165s 2024-01-15T14:23:59.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~165s 2024-01-15T14:23:59.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~165s 2024-01-15T14:23:59.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~165s 2024-01-15T14:24:03.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:24:03.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:24:04.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~170s 2024-01-15T14:24:04.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~170s 2024-01-15T14:24:04.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~170s 2024-01-15T14:24:04.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~170s 2024-01-15T14:24:04.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~170s 2024-01-15T14:24:04.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~170s 2024-01-15T14:24:04.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~170s 2024-01-15T14:24:04.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~170s 2024-01-15T14:24:04.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~170s 2024-01-15T14:24:04.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~170s 2024-01-15T14:24:04.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~170s 2024-01-15T14:24:04.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~170s 2024-01-15T14:24:04.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~170s 2024-01-15T14:24:04.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~170s 2024-01-15T14:24:04.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~170s 2024-01-15T14:24:04.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~170s 2024-01-15T14:24:04.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~170s 2024-01-15T14:24:07.036 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:24:09.074 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~175s 2024-01-15T14:24:09.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~175s 2024-01-15T14:24:09.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~175s 2024-01-15T14:24:09.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~175s 2024-01-15T14:24:09.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~175s 2024-01-15T14:24:09.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~175s 2024-01-15T14:24:09.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~175s 2024-01-15T14:24:09.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~175s 2024-01-15T14:24:09.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~175s 2024-01-15T14:24:09.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~175s 2024-01-15T14:24:09.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~175s 2024-01-15T14:24:09.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~175s 2024-01-15T14:24:09.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~175s 2024-01-15T14:24:09.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~175s 2024-01-15T14:24:09.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~175s 2024-01-15T14:24:09.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~175s 2024-01-15T14:24:09.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~175s 2024-01-15T14:24:14.084 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~180s 2024-01-15T14:24:14.084 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~180s 2024-01-15T14:24:14.084 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~180s 2024-01-15T14:24:14.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~180s 2024-01-15T14:24:14.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~180s 2024-01-15T14:24:14.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~180s 2024-01-15T14:24:14.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~180s 2024-01-15T14:24:14.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~180s 2024-01-15T14:24:14.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~180s 2024-01-15T14:24:14.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~180s 2024-01-15T14:24:14.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~180s 2024-01-15T14:24:14.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~180s 2024-01-15T14:24:14.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~180s 2024-01-15T14:24:14.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~180s 2024-01-15T14:24:14.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~180s 2024-01-15T14:24:14.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~180s 2024-01-15T14:24:14.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~180s 2024-01-15T14:24:19.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:24:19.044 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:24:19.089 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~185s 2024-01-15T14:24:19.089 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~185s 2024-01-15T14:24:19.089 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~185s 2024-01-15T14:24:19.089 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~185s 2024-01-15T14:24:19.089 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~185s 2024-01-15T14:24:19.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~185s 2024-01-15T14:24:19.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~185s 2024-01-15T14:24:19.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~185s 2024-01-15T14:24:19.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~185s 2024-01-15T14:24:19.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~185s 2024-01-15T14:24:19.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~185s 2024-01-15T14:24:19.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~185s 2024-01-15T14:24:19.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~185s 2024-01-15T14:24:19.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~185s 2024-01-15T14:24:19.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~185s 2024-01-15T14:24:19.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~185s 2024-01-15T14:24:19.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~185s 2024-01-15T14:24:22.108 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:24:24.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~190s 2024-01-15T14:24:24.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~190s 2024-01-15T14:24:24.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~190s 2024-01-15T14:24:24.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~190s 2024-01-15T14:24:24.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~190s 2024-01-15T14:24:24.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~190s 2024-01-15T14:24:24.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~190s 2024-01-15T14:24:24.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~190s 2024-01-15T14:24:24.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~190s 2024-01-15T14:24:24.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~190s 2024-01-15T14:24:24.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~190s 2024-01-15T14:24:24.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~190s 2024-01-15T14:24:24.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~190s 2024-01-15T14:24:24.095 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~190s 2024-01-15T14:24:24.095 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~190s 2024-01-15T14:24:24.095 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~190s 2024-01-15T14:24:24.095 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~190s 2024-01-15T14:24:25.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:24:25.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:24:25.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:24:25.180 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:24:25.180 INFO:tasks.ceph:Shutting down logrotate 2024-01-15T14:24:25.181 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 257, in ceph_log logrotater.end() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 222, in end self.thread.get() File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 184, in invoke_logrotate procs = ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:24:25.203 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-15T14:24:25.213 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:24:25.225 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-15T14:24:25.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:24:25.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:24:28.284 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:24:29.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~195s 2024-01-15T14:24:29.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~195s 2024-01-15T14:24:29.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~195s 2024-01-15T14:24:29.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~195s 2024-01-15T14:24:29.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~195s 2024-01-15T14:24:29.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~195s 2024-01-15T14:24:29.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~195s 2024-01-15T14:24:29.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~195s 2024-01-15T14:24:29.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~195s 2024-01-15T14:24:29.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~195s 2024-01-15T14:24:29.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~195s 2024-01-15T14:24:29.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~195s 2024-01-15T14:24:29.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~195s 2024-01-15T14:24:29.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~195s 2024-01-15T14:24:29.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~195s 2024-01-15T14:24:29.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~195s 2024-01-15T14:24:29.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~195s 2024-01-15T14:24:34.104 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~200s 2024-01-15T14:24:34.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~200s 2024-01-15T14:24:34.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~200s 2024-01-15T14:24:34.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~200s 2024-01-15T14:24:34.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~200s 2024-01-15T14:24:34.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~200s 2024-01-15T14:24:34.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~200s 2024-01-15T14:24:34.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~200s 2024-01-15T14:24:34.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~200s 2024-01-15T14:24:34.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~200s 2024-01-15T14:24:34.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~200s 2024-01-15T14:24:34.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~200s 2024-01-15T14:24:34.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~200s 2024-01-15T14:24:34.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~200s 2024-01-15T14:24:34.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~200s 2024-01-15T14:24:34.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~200s 2024-01-15T14:24:34.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~200s 2024-01-15T14:24:37.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:24:37.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:24:39.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~205s 2024-01-15T14:24:39.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~205s 2024-01-15T14:24:39.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~205s 2024-01-15T14:24:39.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~205s 2024-01-15T14:24:39.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~205s 2024-01-15T14:24:39.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~205s 2024-01-15T14:24:39.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~205s 2024-01-15T14:24:39.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~205s 2024-01-15T14:24:39.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~205s 2024-01-15T14:24:39.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~205s 2024-01-15T14:24:39.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~205s 2024-01-15T14:24:39.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~205s 2024-01-15T14:24:39.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~205s 2024-01-15T14:24:39.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~205s 2024-01-15T14:24:39.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~205s 2024-01-15T14:24:39.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~205s 2024-01-15T14:24:39.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~205s 2024-01-15T14:24:40.348 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:24:44.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~210s 2024-01-15T14:24:44.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~210s 2024-01-15T14:24:44.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~210s 2024-01-15T14:24:44.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~210s 2024-01-15T14:24:44.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~210s 2024-01-15T14:24:44.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~210s 2024-01-15T14:24:44.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~210s 2024-01-15T14:24:44.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~210s 2024-01-15T14:24:44.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~210s 2024-01-15T14:24:44.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~210s 2024-01-15T14:24:44.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~210s 2024-01-15T14:24:44.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~210s 2024-01-15T14:24:44.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~210s 2024-01-15T14:24:44.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~210s 2024-01-15T14:24:44.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~210s 2024-01-15T14:24:44.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~210s 2024-01-15T14:24:44.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~210s 2024-01-15T14:24:49.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~215s 2024-01-15T14:24:49.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~215s 2024-01-15T14:24:49.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~215s 2024-01-15T14:24:49.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~215s 2024-01-15T14:24:49.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~215s 2024-01-15T14:24:49.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~215s 2024-01-15T14:24:49.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~215s 2024-01-15T14:24:49.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~215s 2024-01-15T14:24:49.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~215s 2024-01-15T14:24:49.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~215s 2024-01-15T14:24:49.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~215s 2024-01-15T14:24:49.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~215s 2024-01-15T14:24:49.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~215s 2024-01-15T14:24:49.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~215s 2024-01-15T14:24:49.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~215s 2024-01-15T14:24:49.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~215s 2024-01-15T14:24:49.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~215s 2024-01-15T14:24:52.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:24:52.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:24:54.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~220s 2024-01-15T14:24:54.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~220s 2024-01-15T14:24:54.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~220s 2024-01-15T14:24:54.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~220s 2024-01-15T14:24:54.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~220s 2024-01-15T14:24:54.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~220s 2024-01-15T14:24:54.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~220s 2024-01-15T14:24:54.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~220s 2024-01-15T14:24:54.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~220s 2024-01-15T14:24:54.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~220s 2024-01-15T14:24:54.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~220s 2024-01-15T14:24:54.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~220s 2024-01-15T14:24:54.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~220s 2024-01-15T14:24:54.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~220s 2024-01-15T14:24:54.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~220s 2024-01-15T14:24:54.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~220s 2024-01-15T14:24:54.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~220s 2024-01-15T14:24:55.420 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:24:58.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:24:58.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:24:58.492 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:24:58.492 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:24:58.543 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:24:58.560 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-15T14:24:58.573 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-15T14:24:58.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:24:58.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:24:59.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~225s 2024-01-15T14:24:59.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~225s 2024-01-15T14:24:59.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~225s 2024-01-15T14:24:59.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~225s 2024-01-15T14:24:59.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~225s 2024-01-15T14:24:59.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~225s 2024-01-15T14:24:59.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~225s 2024-01-15T14:24:59.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~225s 2024-01-15T14:24:59.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~225s 2024-01-15T14:24:59.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~225s 2024-01-15T14:24:59.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~225s 2024-01-15T14:24:59.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~225s 2024-01-15T14:24:59.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~225s 2024-01-15T14:24:59.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~225s 2024-01-15T14:24:59.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~225s 2024-01-15T14:24:59.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~225s 2024-01-15T14:24:59.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~225s 2024-01-15T14:25:01.628 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:25:04.144 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~230s 2024-01-15T14:25:04.144 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~230s 2024-01-15T14:25:04.144 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~230s 2024-01-15T14:25:04.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~230s 2024-01-15T14:25:04.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~230s 2024-01-15T14:25:04.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~230s 2024-01-15T14:25:04.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~230s 2024-01-15T14:25:04.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~230s 2024-01-15T14:25:04.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~230s 2024-01-15T14:25:04.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~230s 2024-01-15T14:25:04.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~230s 2024-01-15T14:25:04.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~230s 2024-01-15T14:25:04.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~230s 2024-01-15T14:25:04.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~230s 2024-01-15T14:25:04.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~230s 2024-01-15T14:25:04.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~230s 2024-01-15T14:25:04.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~230s 2024-01-15T14:25:09.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~235s 2024-01-15T14:25:09.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~235s 2024-01-15T14:25:09.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~235s 2024-01-15T14:25:09.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~235s 2024-01-15T14:25:09.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~235s 2024-01-15T14:25:09.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~235s 2024-01-15T14:25:09.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~235s 2024-01-15T14:25:09.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~235s 2024-01-15T14:25:09.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~235s 2024-01-15T14:25:09.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~235s 2024-01-15T14:25:09.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~235s 2024-01-15T14:25:09.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~235s 2024-01-15T14:25:09.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~235s 2024-01-15T14:25:09.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~235s 2024-01-15T14:25:09.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~235s 2024-01-15T14:25:09.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~235s 2024-01-15T14:25:09.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~235s 2024-01-15T14:25:10.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:25:10.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:25:13.692 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:25:14.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~240s 2024-01-15T14:25:14.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~240s 2024-01-15T14:25:14.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~240s 2024-01-15T14:25:14.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~240s 2024-01-15T14:25:14.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~240s 2024-01-15T14:25:14.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~240s 2024-01-15T14:25:14.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~240s 2024-01-15T14:25:14.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~240s 2024-01-15T14:25:14.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~240s 2024-01-15T14:25:14.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~240s 2024-01-15T14:25:14.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~240s 2024-01-15T14:25:14.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~240s 2024-01-15T14:25:14.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~240s 2024-01-15T14:25:14.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~240s 2024-01-15T14:25:14.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~240s 2024-01-15T14:25:14.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~240s 2024-01-15T14:25:14.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~240s 2024-01-15T14:25:19.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~245s 2024-01-15T14:25:19.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~245s 2024-01-15T14:25:19.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~245s 2024-01-15T14:25:19.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~245s 2024-01-15T14:25:19.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~245s 2024-01-15T14:25:19.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~245s 2024-01-15T14:25:19.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~245s 2024-01-15T14:25:19.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~245s 2024-01-15T14:25:19.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~245s 2024-01-15T14:25:19.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~245s 2024-01-15T14:25:19.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~245s 2024-01-15T14:25:19.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~245s 2024-01-15T14:25:19.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~245s 2024-01-15T14:25:19.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~245s 2024-01-15T14:25:19.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~245s 2024-01-15T14:25:19.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~245s 2024-01-15T14:25:19.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~245s 2024-01-15T14:25:24.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~250s 2024-01-15T14:25:24.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~250s 2024-01-15T14:25:24.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~250s 2024-01-15T14:25:24.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~250s 2024-01-15T14:25:24.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~250s 2024-01-15T14:25:24.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~250s 2024-01-15T14:25:24.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~250s 2024-01-15T14:25:24.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~250s 2024-01-15T14:25:24.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~250s 2024-01-15T14:25:24.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~250s 2024-01-15T14:25:24.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~250s 2024-01-15T14:25:24.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~250s 2024-01-15T14:25:24.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~250s 2024-01-15T14:25:24.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~250s 2024-01-15T14:25:24.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~250s 2024-01-15T14:25:24.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~250s 2024-01-15T14:25:24.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~250s 2024-01-15T14:25:25.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:25:25.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:25:28.764 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:25:29.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~255s 2024-01-15T14:25:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~255s 2024-01-15T14:25:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~255s 2024-01-15T14:25:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~255s 2024-01-15T14:25:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~255s 2024-01-15T14:25:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~255s 2024-01-15T14:25:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~255s 2024-01-15T14:25:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~255s 2024-01-15T14:25:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~255s 2024-01-15T14:25:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~255s 2024-01-15T14:25:29.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~255s 2024-01-15T14:25:29.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~255s 2024-01-15T14:25:29.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~255s 2024-01-15T14:25:29.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~255s 2024-01-15T14:25:29.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~255s 2024-01-15T14:25:29.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~255s 2024-01-15T14:25:29.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~255s 2024-01-15T14:25:31.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:25:31.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:25:31.836 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:25:31.836 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:25:31.836 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:25:31.846 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-15T14:25:31.859 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-15T14:25:31.859 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-15T14:25:31.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:25:31.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:25:34.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~260s 2024-01-15T14:25:34.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~260s 2024-01-15T14:25:34.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~260s 2024-01-15T14:25:34.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~260s 2024-01-15T14:25:34.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~260s 2024-01-15T14:25:34.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~260s 2024-01-15T14:25:34.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~260s 2024-01-15T14:25:34.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~260s 2024-01-15T14:25:34.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~260s 2024-01-15T14:25:34.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~260s 2024-01-15T14:25:34.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~260s 2024-01-15T14:25:34.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~260s 2024-01-15T14:25:34.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~260s 2024-01-15T14:25:34.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~260s 2024-01-15T14:25:34.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~260s 2024-01-15T14:25:34.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~260s 2024-01-15T14:25:34.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~260s 2024-01-15T14:25:34.944 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:25:39.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~265s 2024-01-15T14:25:39.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~265s 2024-01-15T14:25:39.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~265s 2024-01-15T14:25:39.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~265s 2024-01-15T14:25:39.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~265s 2024-01-15T14:25:39.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~265s 2024-01-15T14:25:39.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~265s 2024-01-15T14:25:39.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~265s 2024-01-15T14:25:39.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~265s 2024-01-15T14:25:39.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~265s 2024-01-15T14:25:39.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~265s 2024-01-15T14:25:39.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~265s 2024-01-15T14:25:39.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~265s 2024-01-15T14:25:39.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~265s 2024-01-15T14:25:39.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~265s 2024-01-15T14:25:39.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~265s 2024-01-15T14:25:39.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~265s 2024-01-15T14:25:43.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:25:43.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:25:44.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~270s 2024-01-15T14:25:44.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2024-01-15T14:25:44.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~270s 2024-01-15T14:25:44.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~270s 2024-01-15T14:25:44.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~270s 2024-01-15T14:25:44.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~270s 2024-01-15T14:25:44.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~270s 2024-01-15T14:25:44.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~270s 2024-01-15T14:25:44.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~270s 2024-01-15T14:25:44.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~270s 2024-01-15T14:25:44.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~270s 2024-01-15T14:25:44.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~270s 2024-01-15T14:25:44.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~270s 2024-01-15T14:25:44.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~270s 2024-01-15T14:25:44.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~270s 2024-01-15T14:25:44.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~270s 2024-01-15T14:25:44.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~270s 2024-01-15T14:25:47.008 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:25:49.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~275s 2024-01-15T14:25:49.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~275s 2024-01-15T14:25:49.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~275s 2024-01-15T14:25:49.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~275s 2024-01-15T14:25:49.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~275s 2024-01-15T14:25:49.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~275s 2024-01-15T14:25:49.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~275s 2024-01-15T14:25:49.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~275s 2024-01-15T14:25:49.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~275s 2024-01-15T14:25:49.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~275s 2024-01-15T14:25:49.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~275s 2024-01-15T14:25:49.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~275s 2024-01-15T14:25:49.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~275s 2024-01-15T14:25:49.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~275s 2024-01-15T14:25:49.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~275s 2024-01-15T14:25:49.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~275s 2024-01-15T14:25:49.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~275s 2024-01-15T14:25:54.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~280s 2024-01-15T14:25:54.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~280s 2024-01-15T14:25:54.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~280s 2024-01-15T14:25:54.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~280s 2024-01-15T14:25:54.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~280s 2024-01-15T14:25:54.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~280s 2024-01-15T14:25:54.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~280s 2024-01-15T14:25:54.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~280s 2024-01-15T14:25:54.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~280s 2024-01-15T14:25:54.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~280s 2024-01-15T14:25:54.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~280s 2024-01-15T14:25:54.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~280s 2024-01-15T14:25:54.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~280s 2024-01-15T14:25:54.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~280s 2024-01-15T14:25:54.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~280s 2024-01-15T14:25:54.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~280s 2024-01-15T14:25:54.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~280s 2024-01-15T14:25:59.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:25:59.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:25:59.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~285s 2024-01-15T14:25:59.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~285s 2024-01-15T14:25:59.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~285s 2024-01-15T14:25:59.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~285s 2024-01-15T14:25:59.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~285s 2024-01-15T14:25:59.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~285s 2024-01-15T14:25:59.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~285s 2024-01-15T14:25:59.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~285s 2024-01-15T14:25:59.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~285s 2024-01-15T14:25:59.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~285s 2024-01-15T14:25:59.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~285s 2024-01-15T14:25:59.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~285s 2024-01-15T14:25:59.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~285s 2024-01-15T14:25:59.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~285s 2024-01-15T14:25:59.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~285s 2024-01-15T14:25:59.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~285s 2024-01-15T14:25:59.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~285s 2024-01-15T14:26:02.076 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:04.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~290s 2024-01-15T14:26:04.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~290s 2024-01-15T14:26:04.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~290s 2024-01-15T14:26:04.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~290s 2024-01-15T14:26:04.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~290s 2024-01-15T14:26:04.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~290s 2024-01-15T14:26:04.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~290s 2024-01-15T14:26:04.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~290s 2024-01-15T14:26:04.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~290s 2024-01-15T14:26:04.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~290s 2024-01-15T14:26:04.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~290s 2024-01-15T14:26:04.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~290s 2024-01-15T14:26:04.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~290s 2024-01-15T14:26:04.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~290s 2024-01-15T14:26:04.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~290s 2024-01-15T14:26:04.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~290s 2024-01-15T14:26:04.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~290s 2024-01-15T14:26:05.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:05.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:05.148 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:05.148 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:26:05.149 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:26:05.158 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-15T14:26:05.169 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-15T14:26:05.191 INFO:teuthology.task.internal:Duration was 11493.624832 seconds 2024-01-15T14:26:05.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-15T14:26:05.205 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-15T14:26:05.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:05.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:08.284 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:09.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~295s 2024-01-15T14:26:09.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~295s 2024-01-15T14:26:09.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~295s 2024-01-15T14:26:09.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~295s 2024-01-15T14:26:09.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~295s 2024-01-15T14:26:09.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~295s 2024-01-15T14:26:09.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~295s 2024-01-15T14:26:09.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~295s 2024-01-15T14:26:09.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~295s 2024-01-15T14:26:09.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~295s 2024-01-15T14:26:09.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~295s 2024-01-15T14:26:09.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~295s 2024-01-15T14:26:09.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~295s 2024-01-15T14:26:09.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~295s 2024-01-15T14:26:09.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~295s 2024-01-15T14:26:09.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~295s 2024-01-15T14:26:09.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~295s 2024-01-15T14:26:14.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~300s 2024-01-15T14:26:14.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~300s 2024-01-15T14:26:14.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~300s 2024-01-15T14:26:14.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~300s 2024-01-15T14:26:14.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~300s 2024-01-15T14:26:14.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~300s 2024-01-15T14:26:14.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~300s 2024-01-15T14:26:14.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~300s 2024-01-15T14:26:14.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~300s 2024-01-15T14:26:14.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~300s 2024-01-15T14:26:14.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~300s 2024-01-15T14:26:14.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~300s 2024-01-15T14:26:14.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~300s 2024-01-15T14:26:14.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~300s 2024-01-15T14:26:14.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~300s 2024-01-15T14:26:14.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~300s 2024-01-15T14:26:14.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~300s 2024-01-15T14:26:14.229 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-01-15T14:26:14.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:14.230 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:17.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:17.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:17.308 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:17.308 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:26.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:26.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:29.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:29.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:29.372 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:29.372 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:32.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:32.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:32.444 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:32.444 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:26:32.445 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:26:32.458 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-15T14:26:32.471 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-15T14:26:32.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:32.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:35.548 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:41.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:41.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:44.445 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:44.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:44.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:47.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:47.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:47.516 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:47.516 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:26:47.516 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:26:47.525 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:47.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:47.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:50.588 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:26:59.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:59.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:26:59.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:26:59.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:27:02.588 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:27:02.589 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:27:05.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:27:05.593 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:27:05.660 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:27:05.660 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:27:05.661 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:27:05.672 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-15T14:27:05.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:27:05.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:27:08.764 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:27:14.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:27:14.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:27:17.660 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:27:17.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:27:17.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:27:20.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:27:20.665 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:27:20.732 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:27:20.732 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:27:20.732 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:27:20.744 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:27:20.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-15T14:27:20.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:27:32.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:27:32.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:27:35.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:27:38.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:27:38.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:27:38.876 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:27:38.876 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:27:38.877 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:27:38.889 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-15T14:27:38.901 INFO:teuthology.task.internal:Transferring archived files... 2024-01-15T14:27:38.902 DEBUG:teuthology.misc:Transferring archived files from smithi113:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517064/remote/smithi113 2024-01-15T14:27:38.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:27:38.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:27:39.196 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.141 2024-01-15T14:27:41.984 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:27:48.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-15T14:27:48.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:27:50.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:27:50.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:27:51.260 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.141 2024-01-15T14:27:54.044 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:28:03.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-15T14:28:03.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:28:06.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:28:06.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:28:06.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.141 2024-01-15T14:28:09.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:28:09.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-15T14:28:09.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:28:09.404 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.141 2024-01-15T14:28:09.404 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:28:09.404 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi141 2024-01-15T14:28:09.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-15T14:28:09.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:28:12.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:28:12.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:28:12.188 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:28:12.188 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:28:12.188 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:28:12.199 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-15T14:28:12.211 INFO:teuthology.task.internal:Not uploading archives. 2024-01-15T14:28:12.211 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-15T14:28:12.229 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-15T14:28:12.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:28:12.229 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:28:12.476 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.141 2024-01-15T14:28:15.292 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:28:21.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-15T14:28:21.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:28:24.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:28:24.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:28:24.540 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.141 2024-01-15T14:28:27.356 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:28:36.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-15T14:28:36.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:28:39.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:28:39.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:28:39.612 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.141 2024-01-15T14:28:42.428 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:28:42.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-15T14:28:42.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:28:42.688 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.141 2024-01-15T14:28:42.688 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:28:42.688 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi141 2024-01-15T14:28:45.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-15T14:28:45.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:28:45.500 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-15T14:28:45.500 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T14:28:45.500 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:28:45.510 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-15T14:28:45.521 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-15T14:28:45.532 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: ConnectionError: Failed to reconnect to smithi113 2024-01-15T14:28:45.532 INFO:teuthology.nuke:Checking targets against current locks 2024-01-15T14:28:45.560 INFO:teuthology.nuke:Not nuking smithi113.front.sepia.ceph.com because description doesn't match: rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi != None 2024-01-15T14:28:45.583 INFO:teuthology.nuke:Not nuking smithi141.front.sepia.ceph.com because description doesn't match: rishabh-2024-01-15_11:07:32-fs:volumes-rishabh-clone-stats-testing-default-smithi != None 2024-01-15T14:28:45.584 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi113.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLCA/CGEJo18u9e0CDj25sBHw8iC1BuaeVgBQ5ju1ZHJYeitElGPXNnh/2Fwdt41tcUkHPs+17DXH3op1xm/0xM= smithi141.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8dkBsd2zlYWrvj7QYzhElnvOE/iuNKCyODK6mvmlcRLi0HkL8ocHk57GidyxP8BMY8jl31lLozx7XyO3GSprE= 2024-01-15T14:28:45.592 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} duration: 11493.624831914902 failure_reason: 'Test failure: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress)' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=03572511b5684e238e975862f0858046 status: fail success: false 2024-01-15T14:28:45.592 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-15T14:28:45.676 INFO:teuthology.run:FAIL